When reporting forensic evidence in a criminal trial, it is the experts’ duty to present their findings to jurors as clearly as possible, to allow as little chance for misinterpretation or misunderstanding as possible. However, a recent study by Simon A. Cole and Matt Barno for Science and Justice, titled “Probabilistic reporting in criminal cases in the United States: A baseline study,” highlights that in the various fields of pattern-matching, most reports did not account for a vital factor in weighing the evidence: the probability for error.
Uncertainty is the Rule
Because no forensic finding can be a complete certainty, many forensic statisticians propose that results should be presented in a statistically defensible manner –– that is, reported as a probability. Cole and Barno’s study showed that in most cases, results are reported in categorical terms, using terms like match, consistent with, or identical to, which imply more certainty than uncertainty, and thus may influence a juror’s decision.
“What if practitioners do not actually report in the way that disciplinary standards say they should?” Cole and Barno ask in their study. “In contrast to the robust scholarship on how forensic scientists should report, less attention has been paid to how they do report. Accordingly, our study focuses on the following key research questions:
- To what extent are forensic reports in these disciplines consistent with published standards?
- To what extent are forensic reports in these disciplines probabilistic, and, if so, how is probability expressed?”
To that end, the researchers reviewed reports across four different pattern-matching disciplines: friction ridge prints (“fingerprints”), firearms & toolmarks, questioned documents, and shoeprints. They examined each disciplinary standard, and from there determined the type of report –– Identification, Exclusion, or Inconclusive –– and whether the examiner used terminology that adhered to the applicable standard. They then coded whether the report was probabilistic according to this definition: Did the expert’s report assign any probability greater than zero but less than 1 to the alternate hypothesis that the source of the evidence was someone other than the suspect?
A Call To Action
For those advocating more rigorous standards in reporting, the findings are a call to action. Across all four disciplines, the standard method of reporting used categorical terminology rather than probabilistic, and the majority of reports analyzed adhered to their respective disciplinary standard.
Of the reports analyzed, 89% of friction ridge reports, 67% of firearm/toolmark reports, 50% of questioned document reports, and 87% of shoeprint reports used categorical terms instead of probabilistic terms in their reporting. The research mentions that in the few instances where numerical probability was mentioned, it was discussed in a way that tended to discount or dismiss its use at all.
According to Cole and Barno, probabilistic reporting has “nowhere to go but up.”
Using categorical terms and phrases such as “a perfect match” or “identical to” ascribes certainty to results, sorting what should be a wide spectrum of possibilities into a few proverbial boxes. While experts may be able to account for some nuance within these categories, it is important to remember that jurors are laypeople. Another recent CSAFE-funded study provided new understanding into how much jurors can be swayed by forensic evidence, so it is important that they are given information that not only shows the likelihood of a match, but also accounts for the probability of error.
Disciplinary Standards Upheld
Cole and Barno’s study showed that the majority of all reports analyzed adhered to their respective disciplinary standards (74% of friction ridge print reports, 100% of firearm/toolmark reports, 96% of questioned document reports, and 82% of shoeprint reports). But, the fact that each of these standards used categorical terms, instead of probabilistic, indicates that there is room for improvement when it comes to reporting forensic results.
In order to increase the reliability of expert reporting, the authors suggest that experts attempt to assign numerical probabilities when reporting forensic findings. Forensic statisticians offer a variety of statistics for assigning weight to evidence including random match probabilities, probabilities of inclusion, compound probabilities of exclusion, objective/data-based probabilities, and likelihood ratios. Most importantly, experts should encourage their peers to make probabilistic reporting the standard, rather than the rule.
CSAFE was founded to increase statistical and scientific methods within forensic science disciplines, and this piece of research underscores that commitment. Read more about researcher Simon Cole here. To learn more about how CSAFE provides access to training for forensic science, law, and other professionals, visit our website.
