Simple line graph icon with four connected data points, depicted in dark blue on a light background.

Journal: Law, Probability and Risk

Published: 2023

Primary Author: Hina Arora

Secondary Authors: Naomi Kaplan-Damary, Hal Stern

Type: Publication

Research Area: Forensic Statistics

Studying the repeatability and reproducibility of decisions made during forensic examinations is important in order to better understand variation in decisions and establish confidence in procedures. For disciplines that rely on comparisons made by trained examiners such as for latent prints, handwriting, and cartridge cases, it has been recommended that ‘black-box’ studies be used to estimate the reliability and validity of decisions. In a typical black-box study, examiners are asked to judge samples of evidence as they would in practice, and their decisions are recorded; the ground truth about samples is known by the study designers. The design for such studies includes repeated assessments on forensic samples by different examiners and additionally, it is common for a subset of examiners to provide repeated assessments on the same evidence samples. We demonstrate a statistical approach to analyse the data collected across these repeated trials that offers the following advantages: i) we can make joint inference about repeatability and reproducibility while utilizing both the intra-examiner and inter-examiner data, ii) we can account for examiner–sample interactions that may impact the decision-making process. We demonstrate the approach first for continuous outcomes such as where decisions are made on an ordinal scale with many categories. The approach is next applied to binary decisions and results are presented on the data from two black-box studies.

View Full Article

Related Resources

A thick gray wavy line forming an abstract, looping shape on a light gray background.

An Introduction to the Forensic Handwriting Analysis Software handwriter

July 18, 2025

Full Article

Blue shoeprint with a tread pattern on a white background.

Forensic Footwear: A Retrospective of the Development of the MANTIS Shoe Scanning System

July 10, 2025

There currently are no shoe-scanning devices developed in the United States that can operate in a real-world, variable-weather environment in …

Full Article

A green fingerprint icon on a light gray background.

Examiner consistency in perceptions of fingerprint minutia rarity

July 10, 2025

Friction ridge examiners (FREs) identify distinctive features (minutiae) in fingerprints and consider how rare these observed minutiae are in their …

Full Article