CSAFE is committed to leveraging statistical methods developed in one field of application for use in forensic science, as appropriate. Through research methods, CSAFE professionals are assessing reliability of categorical conclusions, investigations of properties of machine learning algorithms, and studies of score-based likelihood ratios to inform multiple domains.
Hal S. Stern
University of California, Irvine
Iowa State University
Additional Team Members
Naomi Kaplan-Damry firstname.lastname@example.org
Steve Lund (NIST)
CSAFE researchers are using traditional logistic models to study the performance characteristics of individual examiners and individual examples, as well as aggregate performance characteristics for the population. We are aiming to learn about the efficiency of individual examiners and about the population of examiners.
In many forensic science disciplines, especially those involving pattern comparisons, the most common approach to analysis of the evidence involves a series of binary or categorical decisions regarding the evidence. For example, in latent print analysis an examiner initially decides about whether the latent print has enough information to make a formal identification, or not enough value (i.e., there is not enough information to perform the comparison). Following this, assuming the print is of value, the examiner will reach a final decision that is again expressed in categorical terms (e.g., identification, inconclusive, exclusion). There is currently considerable discussion about the role of likelihood ratios in the analysis of forensic evidence. The ENFSI guidelines endorse this approach. Ongoing discussion about the next steps in forensic pattern evidence analysis in the United States however suggests maintaining the focus on categorical outcomes, with perhaps more potential outcomes allowed (a 5-point or larger scale). To date evaluations of forensic examiners have focused primarily on binary decisions (did they correctly identify a pair of known matching items?). There is a need for developing statistical approaches to reliability and validity studies using categorical scales.
The presumed setup for this research project is that data has been collected from a number of forensic science examiners on a number of cases or examples. For each examiner–example pair we have the outcome of the analysis (e.g., determination of value, conclusion with respect to source) on a categorical scale. There may also be data available about characteristics of the examiners and about characteristics of the examples. As a starting point for the research we will consider analyses treating each category as a binary response. This would, for example, in the latent print case, correspond to studying the probability of a VID (value for identification) decision (yes/no) and assessing variation in the decision-making process across examiners and examples. This can be done with traditional logistic models or with the closely related item response theory models used in educational testing. Using such models allows one to obtain information about the performance characteristics of individual examiners (and individual examples) as well as aggregate performance characteristics for the population. The next stage of the analysis will consider generalizations of these models to handle the multiple-category variables. This will focus on multinomial models, including those developed by considering underlying latent continuous variables. The aim of these models, like those described above, is to learn about the efficiency of individual examiners and about the population of examiners.
The primary goals of the proposed project are to (1) explore the strengths and weaknesses of score-based likelihood ratios (SLRs) for quantifying the value of evidence from a statistical perspective, (2) explore the strengths and weaknesses of SLRs from the perspective of forensic evidence interpretation, and (3) determine whether it is possible to develop a framework of evidence interpretation which exploits the strengths of SLRs for impression and pattern evidence. This project would greatly benefit the forensic science community by providing those who wish to use SLRs with a list of recognized strengths and weaknesses, with supporting reasons, as well as a framework for expressing conclusions regarding the SLR results.
Score-based likelihood ratios (SLRs) are becoming increasingly popular for analyzing impression and pattern evidence due to the inherent difficulties in computing Bayes Factors. Some researchers have argued against the use of SLRs within a Bayesian decision paradigm for philosophical reasons, often citing a lack of coherence. Additionally, these researchers might argue that SLRs don’t actually approximate a Bayes Factor, and worse still, there is no indication of how far an SLR may be from the corresponding Bayes Factor. Other researchers have argued that there is no issue with using score-based likelihood ratios in a Bayesian decision paradigm as long as that SLR is accompanied by a measure of calibration of the SLR system. Regardless of which viewpoint one takes, the fact remains that very little research has been published on whether or not SLRs have any validity for quantifying the value of forensic evidence. The primary goals of the proposed project are to (1) explore the strengths and weaknesses of SLRs for quantifying the value of evidence from a statistical perspective, (2) explore the strengths and weaknesses of SLRs from the perspective of forensic evidence interpretation, and (3) determine whether it is possible to develop a framework of evidence interpretation which exploits the strengths of SLRs for impression and pattern evidence. Many forensic science researchers and practitioners have a strong desire for quantitative results for impression and pattern evidence to bolster their “subjective” opinions. This project would greatly benefit the forensic science community by providing those who wish to use SLRs with a list of recognized strengths and weaknesses, with supporting reasons, as well as a framework for expressing conclusions regarding the SLR results.
The primary goals of this project are to (1) explore the extent to which violating the assumption of independence affects the performance of the scoring methods and (2) develop machine learning methods for evaluating comparison scores for forensic evidence which can accommodate and/or adjust for the dependency in the data. The proposed research will impact the community by providing more statistically rigorous methods of computing score-based likelihood ratios for impression and pattern evidence.
Pattern and impression evidence results in data that is inherently high-dimensional and difficult to model statistically. Therefore, many researchers have focused on methods of measuring the similarity between two objects instead. This comparison results in a low-dimensional score which is much easier to model. CSAFE researchers have relied on statistical machine learning algorithms to compute the scores. One of the difficulties with these methods is that the pairwise comparison of all the evidential objects results in a set of dependent scores. This is because any of the scores that contain the same object as one of the two in the comparison will be dependent. The difficulty lies in the fact that while machine learning methods do not have any distributional assumptions, most assume independence between the observations in the data. The primary goals of this project are to (1) explore the extent to which violating the assumption of independence affects the performance of the scoring methods and (2) develop machine learning methods for evaluating comparison scores for forensic evidence that can accommodate and/or adjust for the dependency in the data. The proposed research will impact the community by providing more statistically rigorous methods of computing score-based likelihood ratios for impression and pattern evidence. This project builds on the work achieved during the first five years in Project CC, “Statistical and Algorithmic Approaches to Matching Bullets” and in Project EE, “Statistical and Algorithmic Approaches to Shoeprint Analysis,” by critically evaluating the current methods for violations of assumptions and potential areas for correction and improvement before the current methods are deployed in crime labs.
Page 2 of 5
Uncertainty in probabilistic genotyping of low template DNA: A case study comparing STRMix™ and TrueAllele™
Published: 2023 | By: William Thompson
Two probabilistic genotyping (PG) programs, STRMix™ and TrueAllele™, were used to assess the strength of the same item of DNA evidence in a federal criminal case, with strikingly different results. For STRMix, the reported likelihood ratio in favor of the…
Published: 2022 | By: Amanda Luby
This presentation is from the Forensic Big Data Colloquium at the Netherlands Forensic Institute, November 2022. Posted with permission of CSAFE.
Source identification of shoeprints in mock crime scene using an algorithm based on automatic alignment
Published: 2023 | By: Hana Lee
This presentation is from the 75th Anniversary Conference of the American Academy of Forensic Sciences, Orlando, Florida, February 13-18, 2023. Posted with permission of CSAFE
Published: 2023 | By: Frederico Veneri
Machine learning-based Score Likelihood Ratios have been proposed as an alternative to traditional Likelihood Ratios and Bayes Factor to quantify the value of evidence when contrasting two opposing propositions. Under the common source problem, the opposing proposition relates to the…
Published: 2023 | By: Stephanie Reinders
Camera identification addresses the scenario where an investigator has a questioned digital image from an unknown camera. The investigator wants to know whether the questioned image was taken by a camera on a person of interest’s phone. Researchers discovered that…
Published: 2023 | By: Pilhyun (Andrew) Lim
Identifying the source of handwriting is an important application in the field of forensic science that addresses questioned document evidence found in criminal cases and civil litigation. It is difficult, given the idiosyncrasies of a person’s handwriting, to recognize the…
Published: 2023 | By: Anyesha Ray
Questioned Document Examiners (QDEs) are tasked with analyzing handwriting evidence to make source (or writership) determinations. The Center for Statistics and Applications of Forensic Evidence (CSAFE) has previously developed computational methods to automatically extract quantifiable handwriting features and statistical methods…
Published: 2023 | By: Alexandria Arabio
Handwriting comparative analysis is based on the principle that no two individuals can produce the same writing and that an individual cannot exactly reproduce his/her handwriting. This project aims to assess and quantify the natural variations produced by a distinct…
Published: 2023 | By: Kori Khan
The following is from University of California Law San Francisco symposium "Forensic identification in criminal courts," February 2023. Posted with permission of CSAFE.
Published: 2023 | By: Amanda Luby
We investigate two approaches for analyzing spatial coordinate responses using models inspired by Item Response Theory (IRT). In the first, we use a two-stage approach to first construct a pseudoresponse matrix using the spatial information and then apply standard IRT…
Published: 2023 | By: Alicia Carriquiry
This presentation is from World Police Summit 2023, Dubai, United Arab Emirates, March 7-9, 2023. Posted with permission of CSAFE.
An Overview of the Two-Stage, Score-Based Likelihood Ratio, and Bayes Factor Approaches for Writership Determinations
Published: 2023 | By: Danica Ommen
A variety of statistical approaches have been developed at the Center for Statistics and Applications in Forensic Evidence (CSAFE) to address the question of writership for forensic document examinations. Previous work at CSAFE has addressed the closed-set problem, when the…
A Rotation-Based Feature and Bayesian Hierarchical Model for the Forensic Evaluation of Handwriting Evidence in a Closed Set
Published: 2023 | By: Amy Crawford
Forensic handwriting examiners are often tasked with identifying the writer of a particular document. Examples of handwriting evidence include ransom notes, forged documents and signatures, and threatening letters. At present, examiners rely on visual inspection of similarities and differences between…
Published: 2023 | By: Kori Khan
Forensic science plays a critical role in the United States criminal justice system. For decades, many feature-based fields of forensic science, such as firearm and toolmark identification, developed outside the scientific community's purview. The results of these studies are widely…
Published: 2023 | By: Kori Khan
Forensic science plays a critical role in the United States criminal legal system. Historically, however, most feature-based fields of forensic science, including firearms examination and latent print analysis, have not been shown to be scientifically valid. Recently, black-box studies have…
A Comparison of Various Score-Based Likelihood Ratio (SLR) Methods for the Quantitative Assessment of Footwear Evidence
Published: 2023 | By: Valerie Han
This presentation is from the 75th Anniversary Conference of the American Academy of Forensic Sciences, Orlando, Florida, February 13-18, 2023. Posted with permission of CSAFE.
Published: 2022 | By: Tong Zou
In this work, we explore the application of likelihood ratio as a forensic evidence assessment tool to evaluate the causal mechanism of a bloodstain pattern. It is assumed that there are two competing hypotheses regarding the cause of a bloodstain…
Published: 2022 | By: Wangqian Ju
In this paper, we introduce the R package cmpsR, an open-source implementation of the Congruent Matching Profile Segments (CMPS) method developed at the National Institute of Standards and Technology (NIST) for objective comparison of striated tool marks. The functionality of…
Published: 2022 | By: Danica Ommen
Presentation is from the 106th International Association for Identification (IAI) Annual Educational Conference
Measuring Proficiency among Latent Print Examiners: A Statistical Approach from Standardized Testing
Published: 2022 | By: Amanda Luby
This presentation is from the 74th Annual Scientific Conference of the American Academy of Forensic Sciences
Page 2 of 5
Want to collaborate with CSAFE on a project. Contact us to share your idea.