Webinar: Ensemble SLRs for Forensic Evidence Comparison

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Summer 2022 Webinar Series on Thursday, August 25, 2022, from 11:00am-Noon CT. The presentation will be “Ensemble SLRs for Forensic Evidence Comparison.” 

Presenters:
Danica Ommen
Assistant Professor – Department of Statistics, Iowa State University

Federico Veneri Guarch
Graduate Research Assistant – Department of Statistics, Iowa State University

Presentation Description:

To strengthen the statistical foundations of forensic evidence interpretation, likelihood ratios and Bayes factors are advocated to quantify the value of evidence. Both methods rely on formulating a statistical model, which can be challenging for complex evidence. Machine learning score-based likelihood ratios have been proposed as an alternative in those cases. Under this framework, a (dis)similarity score and its distribution under alternative propositions are estimated using pairwise comparisons, but pairwise comparisons of all the evidential objects result in dependent scores. While machine learning methods may not require distributional assumptions, most assume independence. We introduce a sampling and ensembling approach to remedy this lack of independence. We generate sets where assumptions are met to develop multiple score-based  likelihood ratios later aggregated into a final score to quantify the value of evidence.

 

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

Statistical Thinking for Forensic Practitioners Short Course – Session 4

This event is scheduled to take place on Friday, November 4, 2022. A registration form can be found below. 

Session 4 Description:

Session 4: Quantitative Tools for Forensic Evidence Part 2 is the final session in the four-part short course, Statistical Thinking for Forensic Practitioners

The National Academies report in 2009 and the PCAST report in 2016 encouraged consideration of quantitative approaches to assessing forensic evidence. In this session, we review two of the most often-suggested quantitative approaches, the two-stage approach and the likelihood ratio (Bayes factor) approach. Specific topics include:

  • The two-stage approach to assessing forensic evidence
    • The role of statistical tests in assessing the similarity of two samples
    • Approaches for assessing the relevance of observed similarities
  • Introduction to the likelihood ratio approach
    • Definition and interpretation of the likelihood ratio/Bayes factor
    • Possible applications to different types of evidence (DNA, trace, pattern)
    • Score-based likelihood ratios
    • Sensitivity of the likelihood ratio to modeling choices
  • Results of studies comparing different ways of expressing source conclusions

Presenter:
Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:
Session four is the final session in the four-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 3

This event is scheduled to take place on Friday, October 28, 2022. A registration form can be found below. 

Session 3 Description:

Session 3: Quantitative Tools for Forensic Evidence is the third session in the four-part short course, Statistical Thinking for Forensic Practitioners

The National Academies report in 2009 and the PCAST report in 2016 encouraged consideration of quantitative approaches to assessing forensic evidence. In this session, we review two of the most often-suggested quantitative approaches, the two-stage approach and the likelihood ratio (Bayes factor) approach. Specific topics include:

  • The two-stage approach to assessing forensic evidence
    • The role of statistical tests in assessing the similarity of two samples
    • Approaches for assessing the relevance of observed similarities
  • Introduction to the likelihood ratio approach
    • Definition and interpretation of the likelihood ratio/Bayes factor
    • Possible applications to different types of evidence (DNA, trace, pattern)
    • Score-based likelihood ratios
    • Sensitivity of the likelihood ratio to modeling choices
  • Results of studies comparing different ways of expressing source conclusions

Presenter:
Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:
Session three is the third session in the four-part short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in four sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 2

This event is scheduled to take place on Friday, October 21, 2022. A registration form can be found below. 

Session 2 Description:

Session 2: Sampling, Statistics and the Status Quo is the second session in the four-part short course, Statistical Thinking for Forensic Practitioners

Statistical ideas such as sampling, point and interval estimation of population quantities, and hypothesis testing have direct application in forensic science. In this session, we start from fundamental ideas about variability (and its sources) in measurements used in forensic analysis, and discuss methods to ameliorate, quantify, model, and interpret variation and uncertainty in the evaluation of forensic evidence. In more detail, we will:

  • Talk about variability and its sources, and introduce ideas such as reproducibility, repeatability, and accuracy.
  • Briefly talk about populations and samples, and describe some sampling methods useful in forensic applications.
  • Introduce the idea of estimation of population quantities such as means and proportions and of methods to report the uncertainty attached to those estimates.
  • Describe how to carry out a test of hypothesis to compare two means and a test of equivalence to compare two means.
  • Demonstrate how statistical concepts can be used in the study of current forensic science practice (e.g., design and analysis of black box studies)

Presenter:
Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:

Session two is the second session in the four-part short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in four sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

 

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 1

This event is scheduled to take place on Friday, October 14, 2022. A registration form can be found below. 

Session 1 Description:

Probability Concepts and their Relevance to Forensic Science is the first session in the three-session short course, Statistical Thinking for Forensic Practitioners

Probability is the mathematical language of uncertainty. Probabilities are used to describe the frequency or likelihood of events or to characterize measurement uncertainty. In this first session, we introduce the laws of probability and their application in forensic settings. Specific topics include:

  • Definition and interpretation of probability
  • Basic laws of probability
  • Conditional probability and independence of events
  • Bayes’ Theorem and Bayesian statistics

Topics are illustrated with examples drawn from forensic science and relevant legal cases.

Presenter:

Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:

Session 1 is the first session in the four-part short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in four sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 3

This event is scheduled to take place on June, 17, 2022. A registration form can be found below. 

Session 3 Description:

Session 3: Quantitative Tools for Forensic Evidence is the third session in the three-session short course, Statistical Thinking for Forensic Practitioners

The National Academies report in 2009 and the PCAST report in 2016 encouraged consideration of quantitative approaches to assessing forensic evidence. In this session, we review two of the most often-suggested quantitative approaches, the two-stage approach and the likelihood ratio (Bayes factor) approach. Specific topics include:

  • The two-stage approach to assessing forensic evidence
    • The role of statistical tests in assessing the similarity of two samples
    • Approaches for assessing the relevance of observed similarities
  • Introduction to the likelihood ratio approach
    • Definition and interpretation of the likelihood ratio/Bayes factor
    • Possible applications to different types of evidence (DNA, trace, pattern)
    • Score-based likelihood ratios
    • Sensitivity of the likelihood ratio to modeling choices
  • Results of studies comparing different ways of expressing source conclusions

Presenter:
Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:
Session 3 is the third session in the three-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 2

This event is scheduled to take place on June, 10, 2022. A registration form can be found below. 

Session 2 Description:

Session 2: Sampling, Statistics and the Status Quo is the second session in the three-session short course, Statistical Thinking for Forensic Practitioners

Statistical ideas such as sampling, point and interval estimation of population quantities, and hypothesis testing have direct application in forensic science. In this session, we start from fundamental ideas about variability (and its sources) in measurements used in forensic analysis, and discuss methods to ameliorate, quantify, model, and interpret variation and uncertainty in the evaluation of forensic evidence. In more detail, we will:

  • Talk about variability and its sources, and introduce ideas such as reproducibility, repeatability, and accuracy.
  • Briefly talk about populations and samples, and describe some sampling methods useful in forensic applications.
  • Introduce the idea of estimation of population quantities such as means and proportions and of methods to report the uncertainty attached to those estimates.
  • Describe how to carry out a test of hypothesis to compare two means and a test of equivalence to compare two means.
  • Demonstrate how statistical concepts can be used in the study of current forensic science practice (e.g., design and analysis of black box studies)

Presenter:
Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:

Session 2 is the second session in the three-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

 

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 1

This event is scheduled to take place on June, 3, 2022. A registration form can be found below. 

Session 1 Description:

Probability Concepts and their Relevance to Forensic Science is the first session in the three-session short course, Statistical Thinking for Forensic Practitioners

Probability is the mathematical language of uncertainty. Probabilities are used to describe the frequency or likelihood of events or to characterize measurement uncertainty. In this first session, we introduce the laws of probability and their application in forensic settings. Specific topics include:

  • Definition and interpretation of probability
  • Basic laws of probability
  • Conditional probability and independence of events
  • Bayes’ Theorem and Bayesian statistics

Topics are illustrated with examples drawn from forensic science and relevant legal cases.

Presenter:

Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:

Session 1 is the first session in the three-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

Registration:

AAFS 2022 Recap: Understanding Juror Comprehension of Forensic Testimony: Assessing Jurors’ Decision Making and Evidence Evaluation

Empty Courtroom

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence (CSAFE)

 

During the 74th annual American Academy of Forensic Sciences (AAFS) scientific conference, Cassidy Koolmees, a graduate student in legal psychology at Florida International University (FIU), and her colleagues presented a new study entitled Understanding Juror Comprehension of Forensic Testimony: Assessing Jurors’ Decision Making and Evidence Evaluation. This study came after assessing the findings in multiple studies conducted by CSAFE co-director Brandon Garrett et al., who found that a jury’s analysis of forensic testimony was not dependent on the strength of the language used during the testimony. Besides a slight decrease in conviction rates when inconclusive language was used, guilty verdicts remained stable across all conditions that used language indicating a match. This result suggests that the language used to express a match in forensic testimony has little impact on a jury, regardless of the strength of the language or credibility the expert claims.

Building on these findings, Koolmees and her colleagues examined whether jurors could distinguish low-quality testimony from high-quality testimony of forensic experts, using the language guidelines released by the Department of Justice (DOJ) in 2018 as an indicator of quality.

Study participants were put into one of six language-related conditions, where the number of violations of the DOJ language guidelines ranged from zero to five. Participants listened to a full mock trial that included the presentation of forensic evidence. Afterward, they were asked their verdict and how they would rate aspects of the testimony, including confidence in the verdict, clarity of forensic testimony, credibility of the forensic expert, and strength, quality, and usefulness of the forensic evidence.

Most of the dependent variables were found to have no statistical significance between conditions; confidence in the verdict, credibility of the expert, as well as the strength, usefulness, and clarity of the testimony were all consistent across groups.

The only statistically significant difference found between conditions was in the judgment of quality. Only when comparing the conditions of zero guideline violations and four and five violations were there any changes in guilty verdicts, signifying jurors may notice a change in the quality of forensic testimony only when the quality is severely low.

Overall, the study found that, similarly to previous studies, mock jurors are not sensitive to the quality of forensic evidence or to the differences in language used by the experts presenting said evidence. Further research by the FIU group currently being finalized includes versions of the study where jurors are made aware of the DOJ language guidelines before they are presented with expert testimony.

The researchers of the study share CSAFE’s desire for continued education for those involved in criminal trials. Suggestions put forth include simplified jury instructions and a video presentation of instructions. These proposed reforms align with the CSAFE goal of increasing education in forensic evidence for jurors, attorneys, judges, and other relevant parties.

CSAFE supports and oversees substantial contributions to training and education for a wide range of forensic science stakeholders. Explore CSAFE’s available learning opportunities and current training and education research projects at https://forensicstats.org/training-and-education/.

 

Publications referenced by Koolmees in her study:

How Jurors Evaluate Fingerprint Evidence: The Relative Importance of Match Language, Method Information, and Error Acknowledgment
Brandon Garrett and Gregory Mitchell

Mock Jurors’ Evaluation of Firearm Examiner Testimony
Brandon Garrett, Nicholas Scurich and William Crozier

Insights: A Practical Tool for Information Management in Forensic Decisions

INSIGHTS

A Practical Tool for Information Management in Forensic Decisions:

Using Linear Sequential Unmasking-Expanded (LSU-E) in Casework

OVERVIEW

While forensic analysts strive to make their findings as accurate and objective as possible, they are often subject to external and internal factors that might bias their decision making. Researchers funded by CSAFE created a practical tool that laboratories can use to implement Linear Sequential Unmasking-Expanded (LSU-E; Dror & Kukucka, 2021)—an information management framework that analysts can use to guide their evaluation of the information available to them. LSU-E can improve decision quality and reduce bias but, until now, laboratories and analysts have
received little concrete guidance to aid implementation efforts.

Lead Researchers

Quigley-McBride, A.
Dror, I.E.
Roy, T.
Garrett, B.L.
Kukucka, J.

Journal

Forensic Science International: Synergy

Publication Date

17 January 2022

Publication Number

IN 125 IMPL

Goals

1

Identify factors that can bias decision-making.

2

Describe how LSU-E can improve forensic decision processes and conclusions.

3

Present a practical worksheet, as well as examples and training materials, to help laboratories incorporate LSU-E into their casework.

TYPES OF COGNITIVE BIAS

Cognitive biases can emerge from a variety of sources, including:

Figure 1. Eight sources of cognitive bias in forensic science (Dror, 2020)

COGNITIVE BIAS IN FORENSIC SCIENCE

As shown in Figure 1, there are many potential sources of information that can influence analysts’ decisions. Of particular concern is suggestive, task-irrelevant contextual information (such as a suspect’s race, sex, or prior criminal record) that can bias analyst’s decisions in inappropriate ways.

In one famous example, FBI latent print analysts concluded with “100 percent certainty” that a print linked to the 2003 Madrid train bombing belonged to a US lawyer, Brandon Mayfield. It transpired that these analysts were all wrong—that was not Mayfield’s print. Mayfield was Muslim, which might have biased the analysts given the strong, widespread attitudes towards Muslims post 9/11. Also, Mayfield was on the FBI’s “watch list” because he provided legal representation to someone accused of terrorist activities. Combined, these facts led to confirmation bias effects in the analysts’ evaluations and conclusions about Mayfield’s fingerprints.

LSU-E AND INFORMATION
MANAGEMENT

LSU-E is an approach information management which prioritizes case information based on three main criteria:

Biasing power:

How strongly the information might dispose an analyst to a particular conclusion.

Objectivity:

The extent to which the information might be interpreted to have different “meanings” from one analyst to another.

Relevance:

the degree to which the information is essential to the analytic task itself.

IMPLEMENTING LSU-E IN
FORENSICS

Quigley-McBride et al. have created a practical worksheet for laboratories to use when assessing new information.

1

First, the user specifies the information in question and its source

2

Second, they consider the three LSU-E criteria, and rate the information on a scale of 1-5 for each criterion

3

Finally, they describe strategies to minimize any adverse effects the information may have on the decision-making process

Focus on the future

 

Ideally, LSU-E procedures would be applied before the information reaches the analyst. That said, it is still effective when used at any point in the analyst’s workflow and can help analysts become aware of information that can inappropriately influence their work.

In addition to benefits for analysts, implementing LSU-E could help jurors evaluate the reliability of forensic expert testimony. This would not only encourage healthy skepticism among jurors, but could bolster an expert’s credibility by providing documentation of methods used to evaluate and mitigate potential biases in their decisions.