Examiner variability in pattern evidence: proficiency, inconclusive tendency, and reporting styles

The current approach to characterizing uncertainty in pattern evidence disciplines has focused on error rate studies, which provide aggregated error rates over many examiners and pieces of evidence. However, decisions are often not unanimous and error frequency is likely to vary depending on the quality of the evidence and proficiency of the examiner. Item Response Theory, a class of statistical methods used prominently in educational testing, is one promising approach that allows for the estimation of both examiner proficiency and item difficulty, and can also be extended to account for differing response styles. We find that even when examiners largely agree on a final source decision, there is considerable variability in their tendency to make inconclusive decisions and how they report the strength of evidence. We also find similar differences across items, and these differences could have downstream impact on the final assessment of guilt by the trier of fact. Using error rate studies in fingerprint and firearm comparisons, I will review some recent advances, outline challenges in applying IRT in practice, and discuss the implications of these findings.