Skip to content

Statistics

Overarching GOALS

CSAFE is committed to leveraging statistical methods developed in one field of application for use in forensic science, as appropriate. Through research methods, CSAFE professionals are assessing reliability of categorical conclusions, investigations of properties of machine learning algorithms, and studies of score-based likelihood ratios to inform multiple domains.

Hal Stern

Hal S. Stern

Provost and Executive Vice Chancellor and Chancellor's Professor, Co-Director of CSAFE

University of California, Irvine

DanicaOmmen_web

Danica Ommen

Associate Professor

Iowa State University

Additional Team Members

Naomi Kaplan-Damry nkapland@uci.edu

Alicia Carriquiry
alicia@iastate.edu

Heike Hofmann hofmann@iastate.edu

Steve Lund (NIST)
steven.lund@nist.gov

focus Areas

CSAFE researchers are using traditional logistic models to study the performance characteristics of individual examiners and individual examples, as well as aggregate performance characteristics for the population. We are aiming to learn about the efficiency of individual examiners and about the population of examiners.

In many forensic science disciplines, especially those involving pattern comparisons, the most common approach to analysis of the evidence involves a series of binary or categorical decisions regarding the evidence. For example, in latent print analysis an examiner initially decides about whether the latent print has enough information to make a formal identification, or not enough value (i.e., there is not enough information to perform the comparison). Following this, assuming the print is of value, the examiner will reach a final decision that is again expressed in categorical terms (e.g., identification, inconclusive, exclusion). There is currently considerable discussion about the role of likelihood ratios in the analysis of forensic evidence. The ENFSI guidelines endorse this approach. Ongoing discussion about the next steps in forensic pattern evidence analysis in the United States however suggests maintaining the focus on categorical outcomes, with perhaps more potential outcomes allowed (a 5-point or larger scale). To date evaluations of forensic examiners have focused primarily on binary decisions (did they correctly identify a pair of known matching items?). There is a need for developing statistical approaches to reliability and validity studies using categorical scales.  

The presumed setup for this research project is that data has been collected from a number of forensic science examiners on a number of cases or examples. For each examiner–example pair we have the outcome of the analysis (e.g., determination of value, conclusion with respect to source) on a categorical scale. There may also be data available about characteristics of the examiners and about characteristics of the examples. As a starting point for the research we will consider analyses treating each category as a binary response. This would, for example, in the latent print case, correspond to studying the probability of a VID (value for identification) decision (yes/no) and assessing variation in the decision-making process across examiners and examples. This can be done with traditional logistic models or with the closely related item response theory models used in educational testing. Using such models allows one to obtain information about the performance characteristics of individual examiners (and individual examples) as well as aggregate performance characteristics for the population. The next stage of the analysis will consider generalizations of these models to handle the multiple-category variables.  This will focus on multinomial models, including those developed by considering underlying latent continuous variables. The aim of these models, like those described above, is to learn about the efficiency of individual examiners and about the population of examiners.

The primary goals of the proposed project are to (1) explore the strengths and weaknesses of score-based likelihood ratios (SLRs) for quantifying the value of evidence from a statistical perspective, (2) explore the strengths and weaknesses of SLRs from the perspective of forensic evidence interpretation, and (3) determine whether it is possible to develop a framework of evidence interpretation which exploits the strengths of SLRs for impression and pattern evidence. This project would greatly benefit the forensic science community by providing those who wish to use SLRs with a list of recognized strengths and weaknesses, with supporting reasons, as well as a framework for expressing conclusions regarding the SLR results.

Score-based likelihood ratios (SLRs) are becoming increasingly popular for analyzing impression and pattern evidence due to the inherent difficulties in computing Bayes Factors. Some researchers have argued against the use of SLRs within a Bayesian decision paradigm for philosophical reasons, often citing a lack of coherence. Additionally, these researchers might argue that SLRs don’t actually approximate a Bayes Factor, and worse still, there is no indication of how far an SLR may be from the corresponding Bayes Factor. Other researchers have argued that there is no issue with using score-based likelihood ratios in a Bayesian decision paradigm as long as that SLR is accompanied by a measure of calibration of the SLR system. Regardless of which viewpoint one takes, the fact remains that very little research has been published on whether or not SLRs have any validity for quantifying the value of forensic evidence. The primary goals of the proposed project are to (1) explore the strengths and weaknesses of SLRs for quantifying the value of evidence from a statistical perspective, (2) explore the strengths and weaknesses of SLRs from the perspective of forensic evidence interpretation, and (3) determine whether it is possible to develop a framework of evidence interpretation which exploits the strengths of SLRs for impression and pattern evidence. Many forensic science researchers and practitioners have a strong desire for quantitative results for impression and pattern evidence to bolster their “subjective” opinions. This project would greatly benefit the forensic science community by providing those who wish to use SLRs with a list of recognized strengths and weaknesses, with supporting reasons, as well as a framework for expressing conclusions regarding the SLR results.

The primary goals of this project are to (1) explore the extent to which violating the assumption of independence affects the performance of the scoring methods and (2) develop machine learning methods for evaluating comparison scores for forensic evidence which can accommodate and/or adjust for the dependency in the data. The proposed research will impact the community by providing more statistically rigorous methods of computing score-based likelihood ratios for impression and pattern evidence.

Pattern and impression evidence results in data that is inherently high-dimensional and difficult to model statistically. Therefore, many researchers have focused on methods of measuring the similarity between two objects instead. This comparison results in a low-dimensional score which is much easier to model. CSAFE researchers have relied on statistical machine learning algorithms to compute the scores. One of the difficulties with these methods is that the pairwise comparison of all the evidential objects results in a set of dependent scores. This is because any of the scores that contain the same object as one of the two in the comparison will be dependent. The difficulty lies in the fact that while machine learning methods do not have any distributional assumptions, most assume independence between the observations in the data. The primary goals of this project are to (1) explore the extent to which violating the assumption of independence affects the performance of the scoring methods and (2) develop machine learning methods for evaluating comparison scores for forensic evidence that can accommodate and/or adjust for the dependency in the data. The proposed research will impact the community by providing more statistically rigorous methods of computing score-based likelihood ratios for impression and pattern evidence. This project builds on the work achieved during the first five years in Project CC, “Statistical and Algorithmic Approaches to Matching Bullets” and in Project EE, “Statistical and Algorithmic Approaches to Shoeprint Analysis,” by critically evaluating the current methods for violations of assumptions and potential areas for correction and improvement before the current methods are deployed in crime labs.

Knowledge Transfer

  • Type

Found 82 Results
Page 1 of 5

Source Camera Identification with Multi-Camera Smartphones

Type: Research Area(s): ,

Published: 2023 | By: Stephanie Reinders

An overview of source camera identification on multi-camera smartphones, and introduction to the new CSAFE multi-camera smartphone image database, and a summary of recent results on the iPhone 14 Pro's.

View on Digital Repository


An alternative statistical framework for measuring proficiency

Type: Research Area(s): ,

Published: 2023 | By: Amanda Luby

Item Response Theory, a class of statistical methods used prominently in educational testing, can be used to measure LPE proficiency in annual tests or research studies, while simultaneously accounting for varying difficulty among comparisons. Using black box studies in latent…

View on Digital Repository


Examiner variability in pattern evidence: proficiency, inconclusive tendency, and reporting styles

Type: Research Area(s): ,

Published: 2023 | By: Amanda Luby

The current approach to characterizing uncertainty in pattern evidence disciplines has focused on error rate studies, which provide aggregated error rates over many examiners and pieces of evidence. However, decisions are often not unanimous and error frequency is likely to…

View on Digital Repository


Statistical Interpretation and Reporting of Fingerprint Evidence: FRStat Introduction and Overview

Type: , Research Area(s): ,,

Published: 2023 | By: Jeff Salyards

The FRStat is a tool designed to help quantify the strength of fingerprint evidence. Following lengthy development and validation with assistance from CSAFE and NIST, in 2017 the FRStat was implemented at the USACIL. FRStat is now freely available and…

View on Digital Repository


A Gentle Introduction to the Likelihood Ratio: Basic Ideas, Implementation, and Limitations

Type: Research Area(s):

Published: 2023 | By: Alicia Carriquiry

The workshop focuses on the likelihood ratio (LR) approach in forensic science. The LR, a one-number summary, quantifies how well the observations/results are explained by the prosecution's versus the defense’s propositions. While the basic idea behind the LR is simple…

View on Digital Repository


Shoeprint Alignment and Comparison using Maximum Cliques

Type: Research Area(s): ,

Published: 2023 | By: Gautham Venkatasubramanian

This presentation is from the 107th International Association for Identification (IAI) Annual Educational Conference, National Harbor, Maryland, August 20-26, 2023. Posted with permission of CSAFE.

View on Digital Repository


An algorithm for source identification of footwear impressions—its application on pristine shoeprints and crime-scene like shoeprints

Type: Research Area(s): ,

Published: 2023 | By: Hana Lee

This presentation is from the 107th International Association for Identification (IAI) Annual Educational Conference, National Harbor, Maryland, August 20-26, 2023. Posted with permission of CSAFE.

View on Digital Repository


Diagnostic Tools for Automatic Cartridge Case Comparisons

Type: Research Area(s): ,

Published: 2023 | By: Joseph Zemmels

The following was presented at the Association of Firearm and Tool Mark Examiners (AFTE) 2023, Austin, Texas, May 21-26, 2023. Copyright 2023, The Authors. Posted with permission of CSAFE.

View on Digital Repository


Algorithmic assessment of striation similarity between wire cuts

Type: Research Area(s):

Published: 2023 | By: Yuhang Lin

The following was presented at the Association of Firearm and Tool Mark Examiners (AFTE) 2023, Austin, Texas, May 21-26, 2023. Copyright 2023, The Authors. Posted with permission of CSAFE.

View on Digital Repository


Variations and Extensions of Information Leakage Metrics with Applications to Privacy Problems with Imperfect Statistical Information

Type: , Research Area(s): ,

Published: 2023 | By: Shahnewaz Karim Sakib

The conventional information leakage metrics assume that an adversary has complete knowledge of the distribution of the mechanism used to disclose information correlated with the sensitive attributes of a system. The only uncertainty arises from the specific realizations that are…

View on Digital Repository


Camera Device Identification and the Effects of Underexposure

Type: , Research Area(s): ,

Published: 2023 | By: Seth Pierre

Technology today allows a photograph from a digital camera to be matched with the camera that took it. However, the matching software was created over 10 years ago using data that is not necessarily representative of today’s data. The objective…

View on Digital Repository


Shifting decision thresholds can undermine the probative value and legal utility of forensic pattern-matching evidence

Type: Research Area(s): ,

Published: 2023 | By: William Thompson

Forensic pattern analysis requires examiners to compare the patterns of items such as fingerprints or tool marks to assess whether they have a common source. This article uses signal detection theory to model examiners’ reported conclusions (e.g., identification, inconclusive, or…

View on Digital Repository


A statistical approach to aid examiners in the forensic analysis of handwriting

Type: Research Area(s): ,

Published: 2023 | By: Amy Crawford

We develop a statistical approach to model handwriting that accommodates all styles of writing (cursive, print, connected print). The goal is to compute a posterior probability of writership of a questioned document given a closed set of candidate writers. Such…

View on Digital Repository


Ensemble learning for score likelihood ratios under the common source problem

Type: Research Area(s):

Published: 2023 | By: Federico Veneri

Machine learning-based score likelihood ratios (SLRs) have emerged as alternatives to traditional likelihood ratios and Bayes factors to quantify the value of evidence when contrasting two opposing propositions. When developing a conventional statistical model is infeasible, machine learning can be…

View on Digital Repository


A method for quantifying individual decision thresholds of latent print examiners

Type: Research Area(s): ,

Published: 2023 | By: Amanda Luby

In recent years, ‘black box’ studies in forensic science have emerged as the preferred way to provide information about the overall validity of forensic disciplines in practice. These studies provide aggregated error rates over many examiners and comparisons, but errors…

View on Digital Repository


The q–q Boxplot

Type: Research Area(s):

Published: 2021 | By: Jordan Rodu

Boxplots have become an extremely popular display of distribution summaries for collections of data, especially when we need to visualize summaries for several collections simultaneously. The whiskers in the boxplot show only the extent of the tails for most of…

View on Digital Repository


The Contribution of Forensic and Expert Evidence to DNA Exoneration Cases: An Interim Report

Type: , Research Area(s): ,,

Published: 2023 | By: Simon Cole

This report is from Simon A. Cole, Vanessa Meterko, Sarah Chu, Glinda Cooper, Jessica Weinstock Paredes, Maurice Possley, and Ken Otterbourg (2022), The Contribution of Forensic and Expert Evidence to DNA Exoneration Cases: An Interim Report (National Registry of Exonerations…

View on Digital Repository


Likelihood ratios for categorical count data with applications in digital forensics

Type: Research Area(s): ,

Published: 2022 | By: Rachel Longjohn

We consider the forensic context in which the goal is to assess whether two sets of observed data came from the same source or from different sources. In particular, we focus on the situation in which the evidence consists of…

View on Digital Repository


CSAFE Project Update & ASCLD FRC Collaboration

Type: Research Area(s): ,,,,,

Published: 2022 | By: Jeff Salyards

This presentation highlighted CSAFE's collaboration with the ASCLD FRC Collaboration Hub.

View on Digital Repository


Reliability for Binary and Ordinal Data in Forensics

Type: Research Area(s):

Published: 2022 | By: Hina Arora

Black-box studies are a crucial part of assessing the accuracy and reliability of subjective decisions in forensics. The extant black-box studies have generally had two components. In the first study, examiners are asked to assess forensic samples (typically questioned and…

View on Digital Repository


Page 1 of 5

COMMUNITY CALL-TO-ACTION

Want to collaborate with CSAFE on a project. Contact us to share your idea.