Skip to content

Statistics

Overarching GOALS

CSAFE is committed to leveraging statistical methods developed in one field of application for use in forensic science, as appropriate. Through research methods, CSAFE professionals are assessing reliability of categorical conclusions, investigations of properties of machine learning algorithms, and studies of score-based likelihood ratios to inform multiple domains.

stern-hal

Hal S. Stern

Provost and Executive Vice Chancellor and Chancellor's Professor, Co-Director of CSAFE

University of California, Irvine

danica-brick-jpeg

Danica Ommen

Assistant Professor

Iowa State University

Additional Team Members

Naomi Kaplan-Damry nkapland@uci.edu

Alicia Carriquiry
alicia@iastate.edu

Heike Hofmann hofmann@iastate.edu

Steve Lund (NIST)
steven.lund@nist.gov

focus Areas

CSAFE researchers are using traditional logistic models to study the performance characteristics of individual examiners and individual examples, as well as aggregate performance characteristics for the population. We are aiming to learn about the efficiency of individual examiners and about the population of examiners.

In many forensic science disciplines, especially those involving pattern comparisons, the most common approach to analysis of the evidence involves a series of binary or categorical decisions regarding the evidence. For example, in latent print analysis an examiner initially decides about whether the latent print has enough information to make a formal identification, or not enough value (i.e., there is not enough information to perform the comparison). Following this, assuming the print is of value, the examiner will reach a final decision that is again expressed in categorical terms (e.g., identification, inconclusive, exclusion). There is currently considerable discussion about the role of likelihood ratios in the analysis of forensic evidence. The ENFSI guidelines endorse this approach. Ongoing discussion about the next steps in forensic pattern evidence analysis in the United States however suggests maintaining the focus on categorical outcomes, with perhaps more potential outcomes allowed (a 5-point or larger scale). To date evaluations of forensic examiners have focused primarily on binary decisions (did they correctly identify a pair of known matching items?). There is a need for developing statistical approaches to reliability and validity studies using categorical scales.  

The presumed setup for this research project is that data has been collected from a number of forensic science examiners on a number of cases or examples. For each examiner–example pair we have the outcome of the analysis (e.g., determination of value, conclusion with respect to source) on a categorical scale. There may also be data available about characteristics of the examiners and about characteristics of the examples. As a starting point for the research we will consider analyses treating each category as a binary response. This would, for example, in the latent print case, correspond to studying the probability of a VID (value for identification) decision (yes/no) and assessing variation in the decision-making process across examiners and examples. This can be done with traditional logistic models or with the closely related item response theory models used in educational testing. Using such models allows one to obtain information about the performance characteristics of individual examiners (and individual examples) as well as aggregate performance characteristics for the population. The next stage of the analysis will consider generalizations of these models to handle the multiple-category variables.  This will focus on multinomial models, including those developed by considering underlying latent continuous variables. The aim of these models, like those described above, is to learn about the efficiency of individual examiners and about the population of examiners.

The primary goals of the proposed project are to (1) explore the strengths and weaknesses of score-based likelihood ratios (SLRs) for quantifying the value of evidence from a statistical perspective, (2) explore the strengths and weaknesses of SLRs from the perspective of forensic evidence interpretation, and (3) determine whether it is possible to develop a framework of evidence interpretation which exploits the strengths of SLRs for impression and pattern evidence. This project would greatly benefit the forensic science community by providing those who wish to use SLRs with a list of recognized strengths and weaknesses, with supporting reasons, as well as a framework for expressing conclusions regarding the SLR results.

Score-based likelihood ratios (SLRs) are becoming increasingly popular for analyzing impression and pattern evidence due to the inherent difficulties in computing Bayes Factors. Some researchers have argued against the use of SLRs within a Bayesian decision paradigm for philosophical reasons, often citing a lack of coherence. Additionally, these researchers might argue that SLRs don’t actually approximate a Bayes Factor, and worse still, there is no indication of how far an SLR may be from the corresponding Bayes Factor. Other researchers have argued that there is no issue with using score-based likelihood ratios in a Bayesian decision paradigm as long as that SLR is accompanied by a measure of calibration of the SLR system. Regardless of which viewpoint one takes, the fact remains that very little research has been published on whether or not SLRs have any validity for quantifying the value of forensic evidence. The primary goals of the proposed project are to (1) explore the strengths and weaknesses of SLRs for quantifying the value of evidence from a statistical perspective, (2) explore the strengths and weaknesses of SLRs from the perspective of forensic evidence interpretation, and (3) determine whether it is possible to develop a framework of evidence interpretation which exploits the strengths of SLRs for impression and pattern evidence. Many forensic science researchers and practitioners have a strong desire for quantitative results for impression and pattern evidence to bolster their “subjective” opinions. This project would greatly benefit the forensic science community by providing those who wish to use SLRs with a list of recognized strengths and weaknesses, with supporting reasons, as well as a framework for expressing conclusions regarding the SLR results.

The primary goals of this project are to (1) explore the extent to which violating the assumption of independence affects the performance of the scoring methods and (2) develop machine learning methods for evaluating comparison scores for forensic evidence which can accommodate and/or adjust for the dependency in the data. The proposed research will impact the community by providing more statistically rigorous methods of computing score-based likelihood ratios for impression and pattern evidence.

Pattern and impression evidence results in data that is inherently high-dimensional and difficult to model statistically. Therefore, many researchers have focused on methods of measuring the similarity between two objects instead. This comparison results in a low-dimensional score which is much easier to model. CSAFE researchers have relied on statistical machine learning algorithms to compute the scores. One of the difficulties with these methods is that the pairwise comparison of all the evidential objects results in a set of dependent scores. This is because any of the scores that contain the same object as one of the two in the comparison will be dependent. The difficulty lies in the fact that while machine learning methods do not have any distributional assumptions, most assume independence between the observations in the data. The primary goals of this project are to (1) explore the extent to which violating the assumption of independence affects the performance of the scoring methods and (2) develop machine learning methods for evaluating comparison scores for forensic evidence that can accommodate and/or adjust for the dependency in the data. The proposed research will impact the community by providing more statistically rigorous methods of computing score-based likelihood ratios for impression and pattern evidence. This project builds on the work achieved during the first five years in Project CC, “Statistical and Algorithmic Approaches to Matching Bullets” and in Project EE, “Statistical and Algorithmic Approaches to Shoeprint Analysis,” by critically evaluating the current methods for violations of assumptions and potential areas for correction and improvement before the current methods are deployed in crime labs.

Knowledge Transfer

  • Type

Found 35 Results
Page 1 of 2

Tutorial on Likelihood Ratios with Applications in Digital Forensics

Type: Research Area(s): ,,

This CSAFE webinar was held on September 15, 2022. Presenters: Rachel Longjohn PhD Student - Department of Statistics, University of California, Irvine Dr. Padhraic Smyth Chancellor's Professor - Departments of Statistics and Computer Sciences, University of California, Irvine Presentation Description:…


Ensemble SLRs for Forensic Evidence Comparison

Type: Research Area(s): ,,

This CSAFE webinar was held on August 25, 2022. Presenter: Danica Ommen Assistant Professor – Department of Statistics, Iowa State University Presentation Description: To strengthen the statistical foundations of forensic evidence interpretation, likelihood ratios and Bayes factors are advocated to…


Likelihood Ratios for Categorical Evidence With Applications in Digital Evidence

Type: Research Area(s): ,

Published: 2022 | By: Rachel Longjohn

The following poster was presented at the 74th Annual Scientific Conference of the American Academy of Forensic Sciences (AAFS), Seattle, Washington, February 21-25, 2022.

View on Digital Repository


Statistical support for weight of evidence determinations of handwriting evidence

Type: Research Area(s): ,

Published: 2021 | By: Danica Ommen

Handwriting analysis is conducted through the expertise of Forensic Document Examiners (FDEs) by visually comparing writing samples. Through their training and years of experience, FDEs are able to recognize critical characteristics of writing to evaluate the evidence of writership. In…

View on Digital Repository


Ensemble of SLR Systems for Forensic Evidence

Type: Research Area(s): ,

Published: 2022 | By: Federico Veneri

Score-based likelihood ratios (SLR) have been used as an alternative to feature-based likelihood ratios when the construction of a probabilistic model becomes challenging or infeasible [1]. Although SLR has been shown to provide an alternative way to present a numeric…

View on Digital Repository


Blind Testing in Firearms Examination: Preliminary Results, Benefits, Limitations, and Future Directions

Type: Research Area(s): ,,

This CSAFE webinar was held on June 23, 2022. Presenters: Maddisen Neuman Quality / Research Associate, Houston Forensic Science Center   Presentation Description: Open proficiency tests created by external vendors are used to meet accreditation requirements and demonstrate examiner competence;…


Extracting Case-Specific Data from Validation Studies

Type: Research Area(s): ,,

This CSAFE webinar was held on May 10, 2022. Presenters: Steve Lund Statistical Engineering Division, NIST Hari Iyer Statistical Engineering Division, NIST Presentation Description: The legal and scientific communities agree that any forensic method or algorithm should undergo validation testing…


Shining a Light on Black Box Studies

Type: Research Area(s): ,

This CSAFE webinar was held on April 22, 2022. Presenters: Dr. Kori Khan Iowa State University Dr. Alicia Carriquiry Iowa State University Presentation Description: The American criminal justice system heavily relies on conclusions reached by the forensic science community. In…


Sampling for Forensic Practitioners Short Course

Type: Research Area(s): ,,

This three-session short course took place on March 25, April 1, and April 8, 2022. Recordings of completed sessions and practicum materials can be found below. Presenters: Alicia Carriquiry Director of CSAFE Iowa State University Joseph Zemmels Graduate Research Assistant…


Modeling And iNventory of Tread Impression System (MANTIS): The development, deployment and application of an active footwear data collection system

Type: Research Area(s): ,,

This CSAFE webinar was held on March 24, 2022. Presenters: Dr. Richard Stone Iowa State University Dr. Susan Vanderplas University of Nebraska, Lincoln Presentation Description: This webinar details the development, capabilities and successful deployment of the Modeling And iNventory of…


Modeling covarying responses of forensic decision-makers within an IRT framework

Type: Research Area(s):

Published: 2020 | By: Amanda Luby

Presented at Joint Statistical Meetings (JSM) 2020

View on Digital Repository


Studying Reproducibility and Reliability for Pattern Evidence Comparisons

Type: Research Area(s):

Published: 2021 | By: Hina Arora

Presented at International Association for Identification (IAI) 2021

View on Digital Repository


Statistics in the Pursuit of Justice—A More Principled Strategy to Analyze Forensic Evidence.

Type: Research Area(s):

Published: 2021 | By: Alicia Carriquiry

Florence Nightingale David Lecture at Joint Statistical Meetings (JSM)

View on Digital Repository


Reliability for Ordinal Data in Forensics

Type: Research Area(s):

Published: 2021 | By: Hina Arora

Presented at Joint Statistical Meetings (JSM) 2021

View on Digital Repository


Studying Reliability for Pattern Evidence Comparisons

Type: Research Area(s):

Published: 2021 | By: Hina Arora

Presentation prepared for International Association for Identification (IAI) 2021

View on Digital Repository


Bayesian Item Response Theory for Human Factors in Forensic Science

Type: Research Area(s):

Published: 2021 | By: Amanda Luby

Presented at Joint Statistical Meetings (JSM) 2021

View on Digital Repository


Machine Learning Methods for Dependent Data Resulting from Forensic Evidence Comparisons

Type: Research Area(s):

Published: 2021 | By: Danica Ommen

Presented at the Joint Statistical Meeting in 2021

View on Digital Repository


Using Mixture Models to Examine Group Differences – Studying Juror Perceptions of the Strength of Forensic Science Evidence

Type: Research Area(s): ,

Published: 2021 | By: Naomi Kaplan-Damary

The following presentation was presented at 2021 International Association for Identification (IAI)

View on Digital Repository


Science Bench Book for Judges

Type: Research Area(s):

Published: 2019 | By: Alicia Carriquiry (16 contributing authors)

The National Judicial College and the Justice Speakers Institute, LLC developed this bench book with support from the State Justice Institute to help judges determine the admissibility of evidence based on sound forensic science. The fact is, few judges were…

View on Digital Repository


Using the Likelihood Ratio in Bloodstain Pattern Analysis

Type: Research Area(s): ,

Published: 2021 | By: Daniel Attinger

There is an apparent paradox that the likelihood ratio (LR) approach is an appropriate measure of the weight of evidence when forensic findings have to be evaluated in court, while it is typically not used by bloodstain pattern analysis (BPA)…

View on Digital Repository


Page 1 of 2

COMMUNITY CALL-TO-ACTION

Want to collaborate with CSAFE on a project. Contact us to share your idea.