Redaction accuracy testing means running your own footage through a tool and scoring two things. Recall is the share of real faces it covered. Precision is the share of its redactions that were correct. A vendor's 99% figure tells you neither until you know what it counts.
That figure could describe frames, faces, tracked identities or seconds of video. Each gives a different result on the same clip. A tool can score 99% on frames and still leave one person unblurred for eleven seconds. That is long enough for anyone watching to identify them.
What are recall and precision in redaction?
Take 400 faces in your test footage. If the tool covers 388, recall is 97%. The other 12 are misses, and each one is an identity left visible.
Now count the boxes. Say the tool draws 460 and 388 of them sit on real faces. Precision is 84%. The other 72 boxes land on things that are not faces, such as posters or mannequins. They cost nothing in privacy terms but add reviewer time, and in evidence they can hide something that matters.
Apply both measures to each category separately, such as faces and number plates. A blended score lets a strong category cover for a weak one.
| Measure | What it tells you | Worked example |
|---|---|---|
| Recall | How many real faces were found | 388 of 400 = 97% |
| Precision | How many redactions were correct | 388 of 460 = 84% |
| Misses | Faces left visible | 12 |
| False alarms | Boxes with no face under them | 72 |
Why does one accuracy figure tell you so little?
A single figure hides which unit was counted, which footage was used and which threshold was set. Take the unit first. Frame-level accuracy forgives a face missed for a second as long as the other 29 frames are clean. Track-level accuracy does not, and neither will the person in the footage.
Then the footage. Detection recall falls on video below 720p, in low light, and when faces are small, turned away or behind glass. The size of the drop between clean daylight and a poor night camera depends on the tool. In the illustrative figures later in this article, recall falls from 98% to 89%, a nine point gap. That is why you measure it yourself. A figure from clean daylight 1080p says little about a 480p corridor camera at 2 a.m.
The setting matters most. Every tool lets you trade recall against precision through a confidence threshold. A vendor can quote recall at a sensitive setting and precision at a strict one, and both numbers can be true. They just never describe the same run. Speed claims carry the same problem, which we cover in our look at accuracy versus speed in AI redaction.
If you want to benchmark face detection against a public reference first, WIDER FACE is a public dataset published in 2016. It reports results on easy, medium and hard subsets, so you can see how scores fall as faces get smaller and more obscured. It is made of still photographs, not CCTV or body-worn video, and it does not cover number plates or your export compression. Treat it as a way to read a vendor's claim, and ask which subset a figure came from. It cannot replace your own clips.
How do you build a test set that looks like your footage?
Pull clips from your own archive, not a demo reel. Forty to sixty clips of three to five minutes each give you seven to ten clips for each of six camera and lighting combinations. That is enough to show where recall changes by camera and lighting, though not to separate gaps of a point or two. It is about two to five hours of footage in total. You want at least 500 labeled faces per category. With 50 faces a single miss moves recall by two points, and with 500 it moves it by 0.2.
Make sure the set includes:
Every camera type you actually use: fixed CCTV, body-worn, dashcam, phone.
Daylight, dusk, night, and glare from windows or headlights.
Crowds, people at the edge of the frame, and very small faces such as a pedestrian across a car park.
Hats, masks, helmets, hoods, sunglasses and people facing away.
Hard negatives: posters, screens showing faces, reflections and mannequins.
Compression at the quality your video system really exports, not a pristine original.
Label the set yourself. Have two people mark faces independently on sampled frames, then settle the disagreements. Keep the labels private. If a vendor sees them, the results measure how well it tuned to your labels. Labelling is the slow part, so plan for it to take longer than the testing.
Which costs more: missed detections or over-redaction?
The two errors do not cost the same. A miss can expose a third party. Article 15(4) of the GDPR says the right to a copy must not adversely affect the rights and freedoms of others. A visible bystander in a subject access response is exactly that. It applies to every subject access request that includes video.
Over-redaction costs differently. Under US FOIA, agencies release reasonably segregable portions of a record, and the exemptions on the official FOIA portal are narrow. Blurring a face that no exemption covers invites an administrative appeal. Each redaction must tie to an exemption, such as Exemption 6 for personal privacy or Exemption 7(C) for law enforcement records. It also strips meaning from the footage. Blur every face in a crowd and a witness statement no longer matches the video.
Choose the threshold by what happens to the output. For public release or disclosure to a third party, favor recall and budget reviewer time for the false alarms. For internal review that never leaves the building, you can accept more misses.
One limit applies to every tool. No automated system reaches 100% recall on difficult footage. Plan a human review pass for anything leaving your organization, and time it. At 72 false alarms per 400 faces, that pass gets long.
How do you run redaction accuracy testing in five steps?
Fix the settings. Run each tool at its default and at its most sensitive setting, and write both down. Compare recall at the two settings. A gap of more than 5 points is a rough flag, not a standard. On 400 faces it means 20 more misses at the default, so the tool depends heavily on one configuration. Note that in your results.
Use the same clips and export quality. Send identical files through every tool, including Secure Redact video redaction if it is on your shortlist. Ask each vendor for a trial so you run the files yourself.
Score against your labels. Count a face as found only if the redaction covers it from forehead to chin in the sampled frame. A partial box counts as a miss.
Measure exposure time. For every miss, count how many seconds the face stays visible. One frame at 30 frames per second lasts 0.03 seconds. Eleven seconds is a disclosure.
Time the fix. Record how long a reviewer needs to correct each tool's output, since that is the real cost of low precision.
If you have hundreds of clips, a redaction API lets you script the runs instead of uploading by hand. Log which tool version and settings produced each output, the way a chain of custody record would. Keep the log alongside your results so you can show how you reached your decision if it is challenged.
Read results per category and condition
Report recall and precision for each category under each condition. A useful result looks like this: daylight at 1080p, recall 98% and precision 91%; night at 480p, recall 89% and precision 78%. Those numbers are invented for illustration. The gap between the rows is the information you were after.
If a vendor will not run your clips or will not state the threshold behind its figures, treat the headline number as marketing. If the weak row matches your worst camera, you now know where human review has to concentrate.
Start small this week. Pick the single worst camera in your estate, pull 20 clips, and label them. Run them at default and at the most sensitive setting, and you will have a recall and precision pair for your hardest footage.
Frequently asked questions
-
Recall is the share of faces or other personal data in the footage that the tool actually redacted. If the footage holds 400 faces and the tool covers 388, recall is 97%, and the 12 it missed are identities left visible. Measure it separately for each category and lighting condition, because one blended figure can hide a weak spot.
-
Aim for 40 to 60 clips and at least 500 labeled faces per category. With only 50 faces, one miss moves recall by two points, so small sets give noisy results. Include every camera type and lighting condition you actually use.
-
Not on its own. The figure may count frames rather than faces, come from clean daylight footage, or reflect a threshold you would never run. Ask for recall and precision per category and per condition, then check them on your own clips.
-
It depends on where the output goes. For public release or disclosure to third parties, favor recall and plan reviewer time for false alarms. For internal review that stays inside your organization, you can accept lower recall in exchange for fewer boxes to check.