Forensic science disciplines such as latent print examination, bullet and cartridge case comparisons, and shoeprint analysis, involve subjective decisions by forensic experts throughout the examination process. Most of the decisions involve ordinal categories. Examples include a three-category outcome for latent print comparisons (exclusion, inconclusive, identification) and a seven-category outcome for footwear comparisons (exclusion, indications of non-association, inconclusive, limited association of class characteristics, association of class characteristics, high degree of association, identification). As the results of the forensic examinations of evidence can heavily influence the outcomes of court proceedings, it is important to assess the reliability and accuracy of the underlying decisions. “Black box” studies are the most common approach for assessing the reliability and accuracy of subjective decisions. In these studies, researchers produce evidence samples consisting of a sample of questioned source and a sample of known source where the ground truth (same source or different source) is known. Examiners provide assessments for selected samples using the same approach they would use in actual casework. These studies often have two phases; the first phase comprises of decisions on samples of varying complexities by different examiners, and the second phase involves repeated decisions by the same examiner on a (usually) small subset of samples that were encountered by examiners in the first phase. We provide a statistical method to analyze ordinal decisions from black-box trials with the objective of obtaining inferences for the reliability of these decisions and quantifying the variation in decisions attributable to the examiners, the samples, and statistical interaction effects between examiners and samples. We present simulation studies to judge the performance of the model on data with known parameter values and apply the model to data from a handwritten signature complexity study, a latent fingerprint examination black-box study, and a handwriting comparisons black-box study
Reliability of ordinal outcomes in forensic black-box studies
Journal: Forensic Science International
Published: 2024
Primary Author: Hina M. Arora
Secondary Authors: Naomi Kaplan-Damary, Hal Stern
Type: Publication
Research Area: Forensic Statistics
Related Resources
Score-based Likelihood Ratios Using Stylometric Text Embeddings
We consider the problem setting in which we have two sets of texts in digital form and would like to quantify our beliefs that the two sets of texts were…
Statistics and its Applications in Forensic Science and the Criminal Justice System
This presentation is from the 2024 Joint Statistical Meetings (JSM), Portland, Oregon, August 3-8, 2024.
Algorithmic matching of striated tool marks
Automatic matching algorithms for assessing the similarity between striation marks have been investigated for bullet lands and some tool marks, such as screwdrivers. We are interested in the investigation of…
Silencing the Defense Expert
In the wake of the 2009 NRC and 2016 PCAST Reports, the Firearms and Toolmark (FATM) discipline has come under increasing scrutiny. Validation studies like AMES I, Keisler, AMES II,…