A key FBI photo analysis method has serious flaws: study

A study published this week casts doubt on the reliability of a technique the FBI Laboratory has used for decades to identify criminals by purporting to match their bluejeans with those photographed in surveillance images, potentially undermining evidence used to win numerous convictions.

The FBI’s method, used principally in bank robbery cases, matches denim pants by the light and dark patches along their seams, called wear marks. An FBI examiner’s scientific journal article on bluejeans identification in 1999 argued that wear marks create, effectively, a barcode that is unique on every pair. That article provided a legal foundation for the FBI to use an array of similar techniques to assert matches for clothes, vehicles, human faces and skin features.

After a ProPublica investigation raised questions about the technique, Hany Farid, a University of California, Berkeley, computer science professor and leading forensic image analyst, and Sophie Nightingale, a postdoctoral researcher in image science, tested the bureau’s method and found several serious flaws. Their study, published this week in the journal Proceedings of the National Academy of Sciences, is the first known independent research on the technique’s reliability, even though the courts have allowed bluejeans identifications as trial evidence for years.

The new study determined that seams on different pairs of bluejeans are often highly similar. Separately, multiple pictures of the same pant seam, taken under varying conditions, can appear starkly different from one another.

Taken together, the authors write, these deficiencies show “identification based on denim jeans should be used with extreme caution, if at all.”

The FBI declined to comment on the study.

In its articles last year, ProPublica revealed that FBI examiners have tied defendants to crimes in thousands of cases over the past half-century by using crime-scene pictures in unproven ways and, at times, have given jurors baseless statistics to say the risk of errors in their analyses was extremely low. In several cases, the FBI’s most prominent image examiner contradicted the original conclusions and results in his lab reports when presenting evidence to criminal courts, FBI records and legal filings show.

The FBI’s issues with image analysis echo earlier controversies over other forensic techniques. The bureau’s lab technicians and scientists had long testified in court that they could determine what fingertip left a print and which scalp grew a hair “to the exclusion of all others.” Research and exonerations by DNA analysis have repeatedly disproved those claims, and the U.S. Department of Justice no longer permits its forensic scientists to make such unequivocal statements.

ProPublica found that examiners on the Forensic Audio, Video and Image Analysis Unit, based at the FBI Lab in Quantico, Virginia, continue to use similarly flawed methods and to testify to the precision of these methods, according to a review of court records and examiners’ written reports and published articles. At ProPublica’s request, several statisticians and forensic science experts reviewed the unit’s methods. The experts identified numerous instances of examiners overstating their techniques’ precision and said some of their assertions defied logic.

In response to ProPublica’s reporting, Nightingale and Farid said they decided to test the FBI’s photo comparison techniques, starting with bluejeans identification.

The researchers purchased 100 pairs of jeans from local second-hand stores and collected images of more than 100 additional pairs of jeans through Mechanical Turk, the Amazon service that provides workers to complete tasks. The researchers used four high-resolution pictures of the seams on each pant leg.

They documented wear marks in the same manner FBI examiners do. But the researchers used what is known as signal analysis to digitally convert the patterns into numeric values and calculate how similar the jeans in different images were to each other.

The authors were consistently able to mark the same features, suggesting the first step in the bureau’s process works as intended.

But then the analysis measured wear mark patterns and found the FBI Lab’s method struggled to match images of the same pant seam, which were frequently no more similar to one another than to seams from different pairs.

Nightingale and Farid hypothesize that denim jeans are too flexible, as the material easily stretches and shrinks, changing how wear marks appear, even moment to moment.

The technique failed to correctly match images of the same bluejeans in most cases unless they allowed for a high rate of false positives. When inaccurate matches were limited to one in 10,000, it identified less than 30% of the true matches.

Ultimately, comparing bluejeans seams is relatively useless, Farid said. “If you’re willing to tolerate that only one in four times this will be useful, OK, fine, use the analysis.”

Brandon Garrett, a Duke University law professor who studies the reliability of forensic science, agreed the study’s results cast serious doubt on the accuracy of jeans identifications, similar to the problems earlier research found in hair fiber and tool mark evidence.

“This is one of many studies uncovering non-trivial error rates for forensic techniques,” Garrett said. “Any lawyer or any judge in a case involving this discipline should, at minimum, hear about the error rates. Many people assume that these techniques are perfect.”

The error rates found in the study are probably the best-case scenario, the researchers said. Every image used in the study was taken in a controlled setting, under good lighting and with the pant seams flattened against a hard surface.

FBI examiners often analyze low-quality images from security cameras and “it is reasonable to expect that the reliability of this technique may degrade under real-world imaging conditions,” the authors wrote.

They argue that all image pattern analysis should undergo validation tests, performed by researchers independent of the FBI and other forensic laboratories. “Mistakes in these identifications are costly, resulting in an innocent person being accused or sentenced and a guilty person walking free.”

While further research is critical, Garrett argued that alone isn’t sufficient. He said this study and scores of others make clear the federal government should regulate the work of forensic scientists in the same manner they do clinical laboratories, setting rules and constantly testing their accuracy.

“We’ve known about the need for national regulation for over a decade now,” Garrett said, “and we haven’t seen it.”