Skip to content

Hierarchical Bayesian non-response models for error rates in forensic black-box studies

Journal: Philosophical Transactions of the Royal Society, Mathematical, Physical and Engineering Sciences
Published: 2023
Primary Author: Kori Khan
Secondary Authors: Alicia Carriquiry

Forensic science plays a critical role in the United States criminal legal system. Historically, however, most feature-based fields of forensic science, including firearms examination and latent print analysis, have not been shown to be scientifically valid. Recently, black-box studies have been proposed as a means of assessing whether these feature-based disciplines are valid, at least in terms of accuracy, reproducibility and repeatability. In these studies, forensic examiners frequently either do not respond to every test item or select an answer equivalent to ‘don’t know’. Current black-box studies do not account for these high levels of missingness in statistical analyses. Unfortunately, the authors of black-box studies typically do not share the data necessary to meaningfully adjust estimates for the high proportion of missing responses. Borrowing from work in the context of small area estimation, we propose the use of hierarchical Bayesian models that do not require auxiliary data to adjust for non-response. Using these models, we offer the first formal exploration of the impact that missingness is playing in error rate estimations reported in black-box studies. We show that error rates currently reported as low as 0.4% could actually be at least 8.4% in models accounting for non-response where inconclusive decisions are counted as correct, and over 28% when inconclusives are counted as missing responses. These proposed models are not the answer to the missingness problem in black-box studies. But with the release of auxiliary information, they can be the foundation for new methodologies to adjust for missingness in error rate estimations.  This article is part of the theme issue ‘Bayesian inference: challenges, perspectives, and prospects’.

Related Resources

The q–q Boxplot

The q–q Boxplot

Boxplots have become an extremely popular display of distribution summaries for collections of data, especially when we need to visualize summaries for several collections simultaneously. The whiskers in the boxplot…
Forensic Science in Legal Education

Forensic Science in Legal Education

In criminal cases, forensic science reports and expert testimony play an increasingly important role in adjudication. More states now follow a federal reliability standard, following Daubert v. Merrell Dow Pharmaceuticals…
The Contribution of Forensic and Expert Evidence to DNA Exoneration Cases: An Interim Report

The Contribution of Forensic and Expert Evidence to DNA Exoneration Cases: An Interim Report

This report is from Simon A. Cole, Vanessa Meterko, Sarah Chu, Glinda Cooper, Jessica Weinstock Paredes, Maurice Possley, and Ken Otterbourg (2022), The Contribution of Forensic and Expert Evidence to…
Likelihood ratios for categorical count data with applications in digital forensics

Likelihood ratios for categorical count data with applications in digital forensics

We consider the forensic context in which the goal is to assess whether two sets of observed data came from the same source or from different sources. In particular, we…