Statistical Thinking for Forensic Practitioners Short Course – Session 3

This event is scheduled to take place on June, 17, 2022. A registration form can be found below. 

Session 3 Description:

Session 3: Quantitative Tools for Forensic Evidence is the third session in the three-session short course, Statistical Thinking for Forensic Practitioners

The National Academies report in 2009 and the PCAST report in 2016 encouraged consideration of quantitative approaches to assessing forensic evidence. In this session, we review two of the most often-suggested quantitative approaches, the two-stage approach and the likelihood ratio (Bayes factor) approach. Specific topics include:

  • The two-stage approach to assessing forensic evidence
    • The role of statistical tests in assessing the similarity of two samples
    • Approaches for assessing the relevance of observed similarities
  • Introduction to the likelihood ratio approach
    • Definition and interpretation of the likelihood ratio/Bayes factor
    • Possible applications to different types of evidence (DNA, trace, pattern)
    • Score-based likelihood ratios
    • Sensitivity of the likelihood ratio to modeling choices
  • Results of studies comparing different ways of expressing source conclusions

Presenter:
Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:
Session 3 is the third session in the three-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 2

This event is scheduled to take place on June, 10, 2022. A registration form can be found below. 

Session 2 Description:

Session 2: Sampling, Statistics and the Status Quo is the second session in the three-session short course, Statistical Thinking for Forensic Practitioners

Statistical ideas such as sampling, point and interval estimation of population quantities, and hypothesis testing have direct application in forensic science. In this session, we start from fundamental ideas about variability (and its sources) in measurements used in forensic analysis, and discuss methods to ameliorate, quantify, model, and interpret variation and uncertainty in the evaluation of forensic evidence. In more detail, we will:

  • Talk about variability and its sources, and introduce ideas such as reproducibility, repeatability, and accuracy.
  • Briefly talk about populations and samples, and describe some sampling methods useful in forensic applications.
  • Introduce the idea of estimation of population quantities such as means and proportions and of methods to report the uncertainty attached to those estimates.
  • Describe how to carry out a test of hypothesis to compare two means and a test of equivalence to compare two means.
  • Demonstrate how statistical concepts can be used in the study of current forensic science practice (e.g., design and analysis of black box studies)

Presenter:
Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:

Session 2 is the second session in the three-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

 

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 1

This event is scheduled to take place on June, 3, 2022. A registration form can be found below. 

Session 1 Description:

Probability Concepts and their Relevance to Forensic Science is the first session in the three-session short course, Statistical Thinking for Forensic Practitioners

Probability is the mathematical language of uncertainty. Probabilities are used to describe the frequency or likelihood of events or to characterize measurement uncertainty. In this first session, we introduce the laws of probability and their application in forensic settings. Specific topics include:

  • Definition and interpretation of probability
  • Basic laws of probability
  • Conditional probability and independence of events
  • Bayes’ Theorem and Bayesian statistics

Topics are illustrated with examples drawn from forensic science and relevant legal cases.

Presenter:

Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:

Session 1 is the first session in the three-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

Registration:

Webinar: Extracting Case-Specific Information from Validation Studies

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Tuesday, May 10, 2022, from 11:00am-Noon CST. The presentation will be “Extracting Case-Specific Information from Validation Studies.” 

Presenters:
Steve Lund
Mathematical Statistician – National Institute of Standards and Technology
Hari Iyer
Mathematical Statistician – National Institute of Standards and Technology

Presentation Description:
Forensic disciplines often summarize validation studies using average error rates. However, almost every forensic discipline has factors that affect the difficulty of a given case (e.g., quantity and quality of a questioned impression or sample), and average performance metrics fail to reflect the difficulty of the current case. This talk presents an approach to characterize the information a set of validation data provides about method performance in a given case.

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

 

Webinar: Shining a Light on Black Box Studies

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Friday, April 22, 2022, from 11:00am-Noon CST. The presentation will be “Shining a Light on Black Box Studies.” 

Presenters:
Dr. Kori Khan
Assistant Professor – Iowa State University
Dr. Alicia Carriquiry
Director – CSAFE

Presentation Description:
The American criminal justice system heavily relies on conclusions reached by the forensic science community. In the last ten years, there has been an increased interest in assessing the validity of the methods used to reach such conclusions. For pattern comparison disciplines, however, this can be difficult because the methods employed rely on visual examinations and subjective determinations. Recently, “black box studies” have been put forward as the gold standard for estimating the rate of errors a discipline makes to assist judges in assessing the validity and admissibility of the analysis of forensic evidence. These studies have since been conducted for various disciplines and their findings are used by judges across the country to justify the admission of forensic evidence testimony. These black box studies suffer from flawed experimental designs and inappropriate statistical analyses. We show that these limitations likely underestimate error rates and preclude researchers from making conclusions about a discipline’s error rates. With a view to future studies, we propose minimal statistical criteria for black box studies and describe some of the data that need to be available to plan and implement such studies.

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

 

AAFS 2022 Recap: Understanding Juror Comprehension of Forensic Testimony: Assessing Jurors’ Decision Making and Evidence Evaluation

Empty Courtroom

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence (CSAFE)

 

During the 74th annual American Academy of Forensic Sciences (AAFS) scientific conference, Cassidy Koolmees, a graduate student in legal psychology at Florida International University (FIU), and her colleagues presented a new study entitled Understanding Juror Comprehension of Forensic Testimony: Assessing Jurors’ Decision Making and Evidence Evaluation. This study came after assessing the findings in multiple studies conducted by CSAFE co-director Brandon Garrett et al., who found that a jury’s analysis of forensic testimony was not dependent on the strength of the language used during the testimony. Besides a slight decrease in conviction rates when inconclusive language was used, guilty verdicts remained stable across all conditions that used language indicating a match. This result suggests that the language used to express a match in forensic testimony has little impact on a jury, regardless of the strength of the language or credibility the expert claims.

Building on these findings, Koolmees and her colleagues examined whether jurors could distinguish low-quality testimony from high-quality testimony of forensic experts, using the language guidelines released by the Department of Justice (DOJ) in 2018 as an indicator of quality.

Study participants were put into one of six language-related conditions, where the number of violations of the DOJ language guidelines ranged from zero to five. Participants listened to a full mock trial that included the presentation of forensic evidence. Afterward, they were asked their verdict and how they would rate aspects of the testimony, including confidence in the verdict, clarity of forensic testimony, credibility of the forensic expert, and strength, quality, and usefulness of the forensic evidence.

Most of the dependent variables were found to have no statistical significance between conditions; confidence in the verdict, credibility of the expert, as well as the strength, usefulness, and clarity of the testimony were all consistent across groups.

The only statistically significant difference found between conditions was in the judgment of quality. Only when comparing the conditions of zero guideline violations and four and five violations were there any changes in guilty verdicts, signifying jurors may notice a change in the quality of forensic testimony only when the quality is severely low.

Overall, the study found that, similarly to previous studies, mock jurors are not sensitive to the quality of forensic evidence or to the differences in language used by the experts presenting said evidence. Further research by the FIU group currently being finalized includes versions of the study where jurors are made aware of the DOJ language guidelines before they are presented with expert testimony.

The researchers of the study share CSAFE’s desire for continued education for those involved in criminal trials. Suggestions put forth include simplified jury instructions and a video presentation of instructions. These proposed reforms align with the CSAFE goal of increasing education in forensic evidence for jurors, attorneys, judges, and other relevant parties.

CSAFE supports and oversees substantial contributions to training and education for a wide range of forensic science stakeholders. Explore CSAFE’s available learning opportunities and current training and education research projects at https://forensicstats.org/training-and-education/.

 

Publications referenced by Koolmees in her study:

How Jurors Evaluate Fingerprint Evidence: The Relative Importance of Match Language, Method Information, and Error Acknowledgment
Brandon Garrett and Gregory Mitchell

Mock Jurors’ Evaluation of Firearm Examiner Testimony
Brandon Garrett, Nicholas Scurich and William Crozier

Webinar: Modeling And iNventory of Tread Impression System (MANTIS) – The development, deployment and application of an active footwear data collection system

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Thursday, March 24, 2022, from 11:00am-Noon CST. The presentation will be “Modeling And iNventory of Tread Impression System (MANTIS): The development, deployment and application of an active footwear data collection system.” 

Presenters:
Dr. Richard Stone
Associate Professor – Iowa State University
Dr. Susan VanderPlas
Assistant Professor – University of Nebraska-Lincoln

Presentation Description:
This session will detail the development, capabilities and successful deployment of the Modeling And iNventory of Tread Impression System (MANTIS). MANTIS Optics Scanner takes real time video of gait as the shoe comes in contact with the cover place (again the clear portion). It synchronizes a series of video cameras to create a detailed image of the shoe that can later be processed by software such as Sift + Ransac to create the tread pattern for comparison. The cameras capture between 8 to 15 megapixels for the configuration below (four cameras located in the housing). The use of video optics is expandable to utilize the laser scanning option, though the current utilization focuses on optical capture, thus allowing for tread capture during dynamic movement, i.e. a person walking or running across the system.

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

 

Webinar: Improving Forensic Decision Making: a Human-Cognitive Perspective

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Thursday, February 17th, 2022, from Noon-1:00 pm CST. The presentation will be “Improving Forensic Decision Making: a Human-Cognitive Perspective.” 

Presenter:
Itiel Dror
Cognitive Neuroscience Researcher – University College London

Presentation Description:
Humans play a critical role in forensic decision making. Drawing upon classic cognitive and psychological research on factors that influence and underpin expert decision making, this webinar will show the weaknesses and vulnerabilities in forensic decision making. The presenter will also propose a broad and versatile approach to strengthening forensic expert decisions.

Associated Reading:
Linear Sequential Unmasking–Expanded (LSU-E): A general approach for improving decision making as well as minimizing noise and bias

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

 

 

Insights: A Practical Tool for Information Management in Forensic Decisions

INSIGHTS

A Practical Tool for Information Management in Forensic Decisions:

Using Linear Sequential Unmasking-Expanded (LSU-E) in Casework

OVERVIEW

While forensic analysts strive to make their findings as accurate and objective as possible, they are often subject to external and internal factors that might bias their decision making. Researchers funded by CSAFE created a practical tool that laboratories can use to implement Linear Sequential Unmasking-Expanded (LSU-E; Dror & Kukucka, 2021)—an information management framework that analysts can use to guide their evaluation of the information available to them. LSU-E can improve decision quality and reduce bias but, until now, laboratories and analysts have
received little concrete guidance to aid implementation efforts.

Lead Researchers

Quigley-McBride, A.
Dror, I.E.
Roy, T.
Garrett, B.L.
Kukucka, J.

Journal

Forensic Science International: Synergy

Publication Date

17 January 2022

Goals

1

Identify factors that can bias decision-making.

2

Describe how LSU-E can improve forensic decision processes and conclusions.

3

Present a practical worksheet, as well as examples and training materials, to help laboratories incorporate LSU-E into their casework.

TYPES OF COGNITIVE BIAS

Cognitive biases can emerge from a variety of sources, including:

Figure 1. Eight sources of cognitive bias in forensic science (Dror, 2020)

COGNITIVE BIAS IN FORENSIC SCIENCE

As shown in Figure 1, there are many potential sources of information that can influence analysts’ decisions. Of particular concern is suggestive, task-irrelevant contextual information (such as a suspect’s race, sex, or prior criminal record) that can bias analyst’s decisions in inappropriate ways.

In one famous example, FBI latent print analysts concluded with “100 percent certainty” that a print linked to the 2003 Madrid train bombing belonged to a US lawyer, Brandon Mayfield. It transpired that these analysts were all wrong—that was not Mayfield’s print. Mayfield was Muslim, which might have biased the analysts given the strong, widespread attitudes towards Muslims post 9/11. Also, Mayfield was on the FBI’s “watch list” because he provided legal representation to someone accused of terrorist activities. Combined, these facts led to confirmation bias effects in the analysts’ evaluations and conclusions about Mayfield’s fingerprints.

LSU-E AND INFORMATION
MANAGEMENT

LSU-E is an approach information management which prioritizes case information based on three main criteria:

Biasing power:

How strongly the information might dispose an analyst to a particular conclusion.

Objectivity:

The extent to which the information might be interpreted to have different “meanings” from one analyst to another.

Relevance:

the degree to which the information is essential to the analytic task itself.

IMPLEMENTING LSU-E IN
FORENSICS

Quigley-McBride et al. have created a practical worksheet for laboratories to use when assessing new information.

1

First, the user specifies the information in question and its source

2

Second, they consider the three LSU-E criteria, and rate the information on a scale of 1-5 for each criterion

3

Finally, they describe strategies to minimize any adverse effects the information may have on the decision-making process

Focus on the future

 

Ideally, LSU-E procedures would be applied before the information reaches the analyst. That said, it is still effective when used at any point in the analyst’s workflow and can help analysts become aware of information that can inappropriately influence their work.

In addition to benefits for analysts, implementing LSU-E could help jurors evaluate the reliability of forensic expert testimony. This would not only encourage healthy skepticism among jurors, but could bolster an expert’s credibility by providing documentation of methods used to evaluate and mitigate potential biases in their decisions.

Webinar: Using Mixture Models to Examine Group Differences: An Illustration Involving the Perceived Strength of Forensic Science Evidence

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Thursday, December 9th, 2021, from 9:00-10:00 am CT. The presentation will be “Using Mixture Models to Examine Group Differences: An Illustration Involving the Perceived Strength of Forensic Science Evidence.” 

Presenter:
Naomi Kaplan Damary, PhD
Lecturer – The Hebrew University of Jerusalem

Presentation Description:
Forensic examiners compare items to assess whether they originate from a common source. In reaching conclusions, they consider the probability of the observed similarities and differences under alternative assumptions regarding the source(s) of the items (i.e., same or different source). These conclusions can be reported in various ways including likelihood ratios or random match probabilities. Thompson et. al., 2018 examined how lay people perceive the strength of these reports through the use of paired comparison models, obtaining rank-ordered lists of the various statements and an indication of the perceived differences among them. The current study expands this research by examining whether the population is comprised of sub-populations that interpret these statements differently and whether their differences can be characterized. A mixture model that allows for multiple sub-populations with possibly different rankings of the statements is fit to the data and the possibility that covariates explain sub-population membership is considered. A deeper understanding of the way potential jurors perceive various forms of forensic reporting could improve communication in the courtroom.

Associated Reading:
Insights: Using Mixture Models to Examine Group Differences Among Jurors

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.