Webinar: Blind Testing in Firearms Examination

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Thursday, June 23, 2022, from 11:00am-Noon CT. The presentation will be “Blind Testing in Firearms Examination: Preliminary Results, Benefits, Limitations, and Future Directions.” 

Presenter:
Maddisen Neuman
Quality / Research Associate – Houston Forensic Science Center

Presentation Description:
Open proficiency tests created by external vendors are used to meet accreditation requirements and demonstrate examiner competence; however, the tests are limited in the ability to monitor the entire quality management system from evidence submission to reporting of results. As such, many scholars have considered blind proficiency testing to be a more precise testing an individual’s accuracy (1).

In December 2015, the Houston Forensic Science Center (HFSC) implemented a blind quality control (blind QC) program in firearms examination. The intent of the blind QC program is to supplement open proficiency tests by creating mock cases that are submitted to the Firearms section as routine casework, with the expectation that the examiners are unaware that they are receiving test cases. Blind QC cases are created by Firearms section management and submitted by members of the Quality Division. Firearms section management evaluates the mock evidence prior to submission to determine the ground truth and expected results. Once the case is complete, the Firearms section management reviews the results to determine satisfactory completion.

This presentation will outline the results of 51 blind QC cases that were reported between December 2015 and June 2021 resulting in 570 sufficiency and comparison conclusions. No eliminations were reported for true matching pairs, and no identifications were reported for true non-matching pairs; thus, all results were considered satisfactory. In 40.3% (n=225) of the comparisons, the ground truth result was either elimination or identification, but an inconclusive conclusion was reached. Variables such as the examiners assigned to the case, training program, examiner experience level, intended complexity of the case, and evidence type were examined for their contributions to the inconclusive determinations. They type of evidence compared was determined to be the main factor contributing to inconclusive results; specifically, bullets were the main contributors to inconclusive results at a rate of 61.8% (n=168) compared to cartridge cases at a rate of 21.5% (n=57).

The webinar will conclude with a discussion on the rate of inconclusive conclusions, benefits and challenges of the blind QC program, and future directions for blind testing research in firearms examination. While a blind testing program can be challenging to implement, HFSC’s blind QC program demonstrates that the quality management system and procedures used by the Firearms section can obtain accurate and reliable results and provides examiners added confidence in court. Additionally, the blind QC program can be tailored to target specific research questions and provide opportunities for collaboration with other laboratories and researchers.

1. Committee on Identifying the Needs of the Forensic Sciences Community, National Research Council. Strengthening forensic science in the United States: A path forward. Washington, DC, 2009.

 

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

 

Statistical Thinking for Forensic Practitioners Short Course – Session 3

This event is scheduled to take place on June, 17, 2022. A registration form can be found below. 

Session 3 Description:

Session 3: Quantitative Tools for Forensic Evidence is the third session in the three-session short course, Statistical Thinking for Forensic Practitioners

The National Academies report in 2009 and the PCAST report in 2016 encouraged consideration of quantitative approaches to assessing forensic evidence. In this session, we review two of the most often-suggested quantitative approaches, the two-stage approach and the likelihood ratio (Bayes factor) approach. Specific topics include:

  • The two-stage approach to assessing forensic evidence
    • The role of statistical tests in assessing the similarity of two samples
    • Approaches for assessing the relevance of observed similarities
  • Introduction to the likelihood ratio approach
    • Definition and interpretation of the likelihood ratio/Bayes factor
    • Possible applications to different types of evidence (DNA, trace, pattern)
    • Score-based likelihood ratios
    • Sensitivity of the likelihood ratio to modeling choices
  • Results of studies comparing different ways of expressing source conclusions

Presenter:
Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:
Session 3 is the third session in the three-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 2

This event is scheduled to take place on June, 10, 2022. A registration form can be found below. 

Session 2 Description:

Session 2: Sampling, Statistics and the Status Quo is the second session in the three-session short course, Statistical Thinking for Forensic Practitioners

Statistical ideas such as sampling, point and interval estimation of population quantities, and hypothesis testing have direct application in forensic science. In this session, we start from fundamental ideas about variability (and its sources) in measurements used in forensic analysis, and discuss methods to ameliorate, quantify, model, and interpret variation and uncertainty in the evaluation of forensic evidence. In more detail, we will:

  • Talk about variability and its sources, and introduce ideas such as reproducibility, repeatability, and accuracy.
  • Briefly talk about populations and samples, and describe some sampling methods useful in forensic applications.
  • Introduce the idea of estimation of population quantities such as means and proportions and of methods to report the uncertainty attached to those estimates.
  • Describe how to carry out a test of hypothesis to compare two means and a test of equivalence to compare two means.
  • Demonstrate how statistical concepts can be used in the study of current forensic science practice (e.g., design and analysis of black box studies)

Presenter:
Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:

Session 2 is the second session in the three-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

 

Registration:

Statistical Thinking for Forensic Practitioners Short Course – Session 1

This event is scheduled to take place on June, 3, 2022. A registration form can be found below. 

Session 1 Description:

Probability Concepts and their Relevance to Forensic Science is the first session in the three-session short course, Statistical Thinking for Forensic Practitioners

Probability is the mathematical language of uncertainty. Probabilities are used to describe the frequency or likelihood of events or to characterize measurement uncertainty. In this first session, we introduce the laws of probability and their application in forensic settings. Specific topics include:

  • Definition and interpretation of probability
  • Basic laws of probability
  • Conditional probability and independence of events
  • Bayes’ Theorem and Bayesian statistics

Topics are illustrated with examples drawn from forensic science and relevant legal cases.

Presenter:

Hal Stern
Co-Director of CSAFE
Provost, Executive Vice Chancellor, and Chancellor’s Professor – University of California, Irvine

About the Short Course:

Session 1 is the first session in the three-session short course, Statistical Thinking for Forensic Practitioners. Dr. Hal Stern introduces fundamental concepts from probability and statistics –– motivated by forensic issues –– followed by a detailed investigation of how they apply to assess forensic evidence’s probative value. This short course will be held online in three sessions. Each session builds upon the previous one(s) and recordings will be available in the event registrants are unable to attend one or more of the live sessions. Researchers, collaborators, and members of the broader forensics and statistics communities are encouraged to attend. Short course registrants who attend all sessions will receive a certificate of completion.

Registration:

Webinar: Extracting Case-Specific Information from Validation Studies

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Tuesday, May 10, 2022, from 11:00am-Noon CST. The presentation will be “Extracting Case-Specific Information from Validation Studies.” 

Presenters:
Steve Lund
Mathematical Statistician – National Institute of Standards and Technology
Hari Iyer
Mathematical Statistician – National Institute of Standards and Technology

Presentation Description:
Forensic disciplines often summarize validation studies using average error rates. However, almost every forensic discipline has factors that affect the difficulty of a given case (e.g., quantity and quality of a questioned impression or sample), and average performance metrics fail to reflect the difficulty of the current case. This talk presents an approach to characterize the information a set of validation data provides about method performance in a given case.

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

 

Webinar: Shining a Light on Black Box Studies

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Friday, April 22, 2022, from 11:00am-Noon CST. The presentation will be “Shining a Light on Black Box Studies.” 

Presenters:
Dr. Kori Khan
Assistant Professor – Iowa State University
Dr. Alicia Carriquiry
Director – CSAFE

Presentation Description:
The American criminal justice system heavily relies on conclusions reached by the forensic science community. In the last ten years, there has been an increased interest in assessing the validity of the methods used to reach such conclusions. For pattern comparison disciplines, however, this can be difficult because the methods employed rely on visual examinations and subjective determinations. Recently, “black box studies” have been put forward as the gold standard for estimating the rate of errors a discipline makes to assist judges in assessing the validity and admissibility of the analysis of forensic evidence. These studies have since been conducted for various disciplines and their findings are used by judges across the country to justify the admission of forensic evidence testimony. These black box studies suffer from flawed experimental designs and inappropriate statistical analyses. We show that these limitations likely underestimate error rates and preclude researchers from making conclusions about a discipline’s error rates. With a view to future studies, we propose minimal statistical criteria for black box studies and describe some of the data that need to be available to plan and implement such studies.

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

 

AAFS 2022 Recap: Understanding Juror Comprehension of Forensic Testimony: Assessing Jurors’ Decision Making and Evidence Evaluation

Empty Courtroom

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence (CSAFE)

 

During the 74th annual American Academy of Forensic Sciences (AAFS) scientific conference, Cassidy Koolmees, a graduate student in legal psychology at Florida International University (FIU), and her colleagues presented a new study entitled Understanding Juror Comprehension of Forensic Testimony: Assessing Jurors’ Decision Making and Evidence Evaluation. This study came after assessing the findings in multiple studies conducted by CSAFE co-director Brandon Garrett et al., who found that a jury’s analysis of forensic testimony was not dependent on the strength of the language used during the testimony. Besides a slight decrease in conviction rates when inconclusive language was used, guilty verdicts remained stable across all conditions that used language indicating a match. This result suggests that the language used to express a match in forensic testimony has little impact on a jury, regardless of the strength of the language or credibility the expert claims.

Building on these findings, Koolmees and her colleagues examined whether jurors could distinguish low-quality testimony from high-quality testimony of forensic experts, using the language guidelines released by the Department of Justice (DOJ) in 2018 as an indicator of quality.

Study participants were put into one of six language-related conditions, where the number of violations of the DOJ language guidelines ranged from zero to five. Participants listened to a full mock trial that included the presentation of forensic evidence. Afterward, they were asked their verdict and how they would rate aspects of the testimony, including confidence in the verdict, clarity of forensic testimony, credibility of the forensic expert, and strength, quality, and usefulness of the forensic evidence.

Most of the dependent variables were found to have no statistical significance between conditions; confidence in the verdict, credibility of the expert, as well as the strength, usefulness, and clarity of the testimony were all consistent across groups.

The only statistically significant difference found between conditions was in the judgment of quality. Only when comparing the conditions of zero guideline violations and four and five violations were there any changes in guilty verdicts, signifying jurors may notice a change in the quality of forensic testimony only when the quality is severely low.

Overall, the study found that, similarly to previous studies, mock jurors are not sensitive to the quality of forensic evidence or to the differences in language used by the experts presenting said evidence. Further research by the FIU group currently being finalized includes versions of the study where jurors are made aware of the DOJ language guidelines before they are presented with expert testimony.

The researchers of the study share CSAFE’s desire for continued education for those involved in criminal trials. Suggestions put forth include simplified jury instructions and a video presentation of instructions. These proposed reforms align with the CSAFE goal of increasing education in forensic evidence for jurors, attorneys, judges, and other relevant parties.

CSAFE supports and oversees substantial contributions to training and education for a wide range of forensic science stakeholders. Explore CSAFE’s available learning opportunities and current training and education research projects at https://forensicstats.org/training-and-education/.

 

Publications referenced by Koolmees in her study:

How Jurors Evaluate Fingerprint Evidence: The Relative Importance of Match Language, Method Information, and Error Acknowledgment
Brandon Garrett and Gregory Mitchell

Mock Jurors’ Evaluation of Firearm Examiner Testimony
Brandon Garrett, Nicholas Scurich and William Crozier

Webinar: Modeling And iNventory of Tread Impression System (MANTIS) – The development, deployment and application of an active footwear data collection system

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Thursday, March 24, 2022, from 11:00am-Noon CST. The presentation will be “Modeling And iNventory of Tread Impression System (MANTIS): The development, deployment and application of an active footwear data collection system.” 

Presenters:
Dr. Richard Stone
Associate Professor – Iowa State University
Dr. Susan VanderPlas
Assistant Professor – University of Nebraska-Lincoln

Presentation Description:
This session will detail the development, capabilities and successful deployment of the Modeling And iNventory of Tread Impression System (MANTIS). MANTIS Optics Scanner takes real time video of gait as the shoe comes in contact with the cover place (again the clear portion). It synchronizes a series of video cameras to create a detailed image of the shoe that can later be processed by software such as Sift + Ransac to create the tread pattern for comparison. The cameras capture between 8 to 15 megapixels for the configuration below (four cameras located in the housing). The use of video optics is expandable to utilize the laser scanning option, though the current utilization focuses on optical capture, thus allowing for tread capture during dynamic movement, i.e. a person walking or running across the system.

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

 

Webinar: Improving Forensic Decision Making: a Human-Cognitive Perspective

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Thursday, February 17th, 2022, from Noon-1:00 pm CST. The presentation will be “Improving Forensic Decision Making: a Human-Cognitive Perspective.” 

Presenter:
Itiel Dror
Cognitive Neuroscience Researcher – University College London

Presentation Description:
Humans play a critical role in forensic decision making. Drawing upon classic cognitive and psychological research on factors that influence and underpin expert decision making, this webinar will show the weaknesses and vulnerabilities in forensic decision making. The presenter will also propose a broad and versatile approach to strengthening forensic expert decisions.

Associated Reading:
Linear Sequential Unmasking–Expanded (LSU-E): A general approach for improving decision making as well as minimizing noise and bias

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.

Sign up on the form below (Chrome & Safari web browsers work the best):

 

 

Webinar: Using Mixture Models to Examine Group Differences: An Illustration Involving the Perceived Strength of Forensic Science Evidence

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2022 Webinar Series on Thursday, December 9th, 2021, from 9:00-10:00 am CT. The presentation will be “Using Mixture Models to Examine Group Differences: An Illustration Involving the Perceived Strength of Forensic Science Evidence.” 

Presenter:
Naomi Kaplan Damary, PhD
Lecturer – The Hebrew University of Jerusalem

Presentation Description:
Forensic examiners compare items to assess whether they originate from a common source. In reaching conclusions, they consider the probability of the observed similarities and differences under alternative assumptions regarding the source(s) of the items (i.e., same or different source). These conclusions can be reported in various ways including likelihood ratios or random match probabilities. Thompson et. al., 2018 examined how lay people perceive the strength of these reports through the use of paired comparison models, obtaining rank-ordered lists of the various statements and an indication of the perceived differences among them. The current study expands this research by examining whether the population is comprised of sub-populations that interpret these statements differently and whether their differences can be characterized. A mixture model that allows for multiple sub-populations with possibly different rankings of the statements is fit to the data and the possibility that covariates explain sub-population membership is considered. A deeper understanding of the way potential jurors perceive various forms of forensic reporting could improve communication in the courtroom.

Associated Reading:
Insights: Using Mixture Models to Examine Group Differences Among Jurors

The webinars are free and open to the public, but researchers, collaborators and members of the broader forensics and statistics communities are encouraged to attend. Each 60-minute webinar will allow for discussion and questions.