A NEW NAME AND A NEW HORIZON: THE WILSON CENTER FOR SCIENCE AND JUSTICE

In recognition of a $5 million dollar grant from the Wilson Foundation, CSAFE partner institution Duke University renamed its Center for Science and Justice the Wilson Center for Science and Justice. Funds from the grant will expand the center’s work in three key areas: accuracy of evidence in criminal cases, the role of equity in criminal outcomes, and the mental and behavioral health treatment needs of people in the justice system. 

CSAFE’s own Co-Director, Brandon Garrett, the L. Neil Williams Jr. Professor of Law and Faculty Director of the center, aims for Duke Law School to build a long-term presence as a national leader in reforming the US criminal justice system. The Center, originally launched in September 2019, plans to encourage and instruct students and faculty on how to bolster legal and scientific data-driven research.  

“Bringing law and science together to prevent injustice has been my life’s work, since I was a young lawyer, and now, in my work with the new Wilson Center for Science and Justice at Duke,” said Professor Garrett. 

The Wilson Center champions the integration of students, faculty, and staff across multiple disciplines, gathering their expertise in law, public policy, medicine, the arts, and sciences, to create a unique lens for criminal justice research. The philosophy of interdisciplinary collaboration driving impactful research, policy, and teaching is what forms the foundation of our educational initiatives at CSAFE.

Strengthening research in the prevention of wrongful convictions, the examination of how jurors evaluate forensic evidence, and the accuracy of evidence in court are primary objectives of the Wilson Center — all of which cannot be achieved without the practice of scientifically valid methods. At its core, CSAFE is committed to the application of verified statistical and scientific techniques to ensure the accuracy of forensic analysis and interpretation and to uphold the fair administration of justice.

With more than just a strong alignment of legal and scientific ideology, CSAFE’s partnership with the Wilson Center for Science and Justice will strengthen the field of forensic science and remedy structural inequalities ingrained in the justice system.

Click here to learn more about CSAFE’s commitment to statistical and scientific data-driven research. 

Insights: Treatment of Inconclusives in the AFTE Range of Conclusions

INSIGHTS

Treatment of Inconclusives in the AFTE Range of Conclusions

OVERVIEW

Several studies have estimated the error rates of firearm examiners in recent years, most of which showed very small error rates overall. However, the actual calculation of these error rates, particularly how each study treats inconclusive results, differed significantly between studies. Researchers funded by CSAFE revisited these studies to see how inconclusive results were treated and how these differences impacted their overall error rate calculations.

Lead Researchers

Heike Hofmann
Susan Vanderplas
Alicia Carriquiry

Journal

Law, Probability and Risk

Publication Date

September 2020

Publication Number

IN 113 FT

THE GOALS

1

Survey various studies that assess the error rates of firearms examinations.

2

Determine the differences in how inconclusives are treated in each study.

3

Identify areas where these studies can be improved.

The Study

Hofmann et al. surveyed the most cited black box studies involving firearm and toolmark analysis. These studies varied in structure, having closed-set or open-set data. They were also conducted in different regions, either in the United States and Canada or in the European Union. The most relevant difference, however, was how each study treated inconclusive results.

All studies used one of three methods to treat inconclusives:

  • Option 1: Exclude the inconclusive from the error rate.
  • Option 2: Include the inconclusive as a correct result.
  • Option 3: Include the inconclusive as an incorrect result.

Key Terms:

Black Box Study: a study that evaluates only the correctness of a participant’s decisions.

Closed-set Study: one in which all known and questioned samples come from the same source.

Open-set Study: one in which the questioned samples may come from outside sources.

Results

Option 1 was deemed inappropriate for accurate error rates. Option 2 was useful for error rates of the individual examiners, while Option 3 reflected the error rates of the process itself.

Examiners tended to lean towards identification over inconclusive or elimination. In addition, they were far more likely to reach an inconclusive with different-source evidence, which should have been an elimination in nearly all cases.

Process errors occurred at higher rates than examiner errors.

Design issues created a bias toward the prosecution, such as closed-set studies where all samples came from the same source, prescreened kit components which inflated the rate of identifications, or multiple known sources which could not quantify a proper error rate for eliminations.

Fig. 1. Sketch of the relationship between ground truth of evidence (dots) and examiners’ decisions (shaded areas). In a perfect scenario dots only appear on the shaded area of the same colour. Any dots on differently coloured backgrounds indicate an error in the examination process.

FOCUS ON THE FUTURE

 

Hofmann et al. propose a fourth option:

  • Include the inconclusive as an elimination.
  • Calculate the error rates for the examiner and the process separately.

While most studies included a bias toward the prosecution, this was not the case for studies conducted in the European Union. Further study is recommended to verify this difference and determine its cause.

Insights: Probabilistic Reporting in Criminal Cases in the United States

INSIGHT

Probabilistic Reporting in Criminal Cases in the United States:

A Baseline Study

OVERVIEW

Forensic examiners are frequently asked to give reports and testimonies in court and there have been calls for them to report their findings probabilistically. Terms like match, consistent with or identical are categorical in nature, not statistical –– they do not communicate the value of the evidence in terms of probability. While there is robust debate over how forensic scientists should report, less attention is paid to how they do report.

Lead Researchers

Simon A. Cole 
Matt Barno

Journal

Science & Justice

Publication Date

September 2020

Publication Number

IN 112 IMPL

Key Research Questions

1

To what extent are forensic reports in these disciplines consistent with published standards?

2

To what extent are forensic reports in these disciplines probabilistic, and, if so, how is probability expressed?

APPROACH AND METHODOLOGY

Data Set

572 transcripts and reports from Westlaw, consultants’ files and proficiency tests using a heterogeneous, opportunistic data collection approach.

What

Researchers reviewed reports across four pattern disciplines:

  • Friction Ridge Prints
  • Firearms & Toolmarks
  • Questioned Documents
  • Shoeprints

How

Using disciplinary standards as a framework, researchers determined the type of report being reviewed and if it used standard terminology. Then, they coded each report both for whether or not it was probabilistic and for the type of language used, such as “same source,” “identified” and “consistent.”

KEY TAKEAWAYS for Practitioners

Across all four disciplines, the prevailing standards for reporting were categorical in nature. The majority of reports analyzed adhered to the reporting standards for their discipline –– but discussion of probability was extremely rare and, even in those cases, frequently used to dismiss the use of probability itself.

reports used categorical terms in their reporting

reports used terms that adhered to their disciplinary standards

reports used probabilistic terms

friction ridge prints
89%
firearms & toolmarks
67%
questioned documents
50%
shoemark
87%
friction ridge prints
74%
firearms & toolmarks
100%
questioned documents
96%
shoemark
82%
friction ridge prints
11%
firearms & toolmarks
33%
questioned documents
50%
shoemark
13%

Focus on the future

 

To increase the probabilistic reporting of forensics results:

1

Incorporate probabilistic reporting into disciplinary standards.

2

Educate practitioners, lawyers, and judges on the reasons for, and importance, of probabilistic reporting.

3

Demand that experts quantify their uncertainty when testifying in court.

Insights: Juror Appraisals of Forensic Science Evidence

INSIGHT

Juror Appraisals of Forensic Science Evidence:

Effects of Proficiency and Cross-examination

OVERVIEW

Researchers conducted two studies to determine how much an examiner’s blind proficiency score affects the jurors’ confidence in their testimonies.

Lead Researchers

William E. Crozier
Jeff Kukucka
Brandon L. Garrett

Journal

Forensic Science International

Publication Date

October 2020

Publication Number

IN 111 IMPL

Key Research Questions

1

Determine how disclosing blind proficiency test results can inform a jury’s decision making.

2

Assess how using these proficiency test results in cross-examination can influence jurors.

APPROACH AND METHODOLOGY

WHO

Two separate groups (1,398 participants in Study 1, and 1,420 in Study 2) read a mock trial transcript in which a forensic examiner provided the central evidence.

What

Evidence: bitemark on a victim’s arm or a fingerprint on the robber’s gun.

Blind Proficiency Scores: the examiner either made zero mistakes in the past year (high proficiency), made six mistakes in the past year (low proficiency), claimed high proficiency without proof (high unproven proficiency), or did not discuss their proficiency at all (control).

How

Participants in both studies were asked to render a verdict, estimate the likelihood of the defendant’s guilt, and provide opinions on the examiner and the evidence.

KEY TAKEAWAYS for Practitioners

1

Stating proficiency scores did influence the participants’ verdicts. In both studies, the examiner presented as having low proficiency elicited fewer convictions than the other examiners.

2

While the high-proficiency examiner did not elicit more convictions than the control in Study 1, they not only got more convictions in Study 2, but also proved to withstand cross-examination better than the other examiners.

3

In both studies, proficiency information influenced the participants’ opinions of the examiners themselves, but not their domain’s methods or evidence.

Focus on the future

 

Despite having lower conviction rates, the low-proficiency examiners were still viewed very favorably and still achieved convictions a majority of the time in both studies (65% and 71% respectively), so fears of an examiner being “burned” by a low-proficiency score are largely overblown.

For defense lawyers to ask about proficiency results, they require access to the information. However, crime laboratories can potentially gain a significant advantage by only disclosing high-proficiency scores. Thus, it is important that such information be disclosed evenly and transparently.

Next Steps

 

The components and data of both studies are available on the Open Science Framework.

Insights: Implementing Blind Proficiency Testing in Forensic Laboratories

INSIGHT

Implementing Blind Proficiency Testing in Forensic Laboratories:

Motivation, Obstacles, and Recommendations

OVERVIEW

Accredited forensic laboratories are required to conduct proficiency testing –– but most rely solely on declared proficiency tests. A 2014 study showed that only 10% of forensic labs in the United States performed blind proficiency testing, whereas blind tests are standard in other fields including medical and drug testing laboratories. Researchers wanted to identify the barriers to widespread blind proficiency testing and generate solutions to removing these obstacles. After reviewing the existing research, they realized they must convene a meeting of experts to establish an understanding of the challenges to implementation.

Lead Researchers

Robin Mejia
Maria Cuellar
Jeff Salyards

Journal

Forensic Science International: Synergy

Publication Date

September 2020

Publication Number

IN 110 LP

Participants

CSAFE met with laboratory directors and quality managers from seven forensic laboratory systems in the eastern US and the Houston Forensic Science Center. Two of the quality managers represented the Association of Forensic Quality Assurance Managers (AFQAM). In addition, several professors, graduate students and researchers from three universities attended the meeting.

APPROACH AND METHODOLOGY

1

Compare blind proficiency testing to declared testing then have participants discuss the potential advantages of establishing blind testing as standard.

2

Facilitate and document a discussion of the logistical and cultural barriers labs might face when adopting blind testing. Use this to create a list of challenges.

3

Collect and analyze suggested steps labs can take to overcome the list of challenges to implementing blind proficiency testing.

Challenges and Solutions

Challenge Proposed Solution
Realistic test case creation can be complex.
Quality managers develop the expertise to create test cases; laboratories create a shared evidence bank.
The development of realistic submission materials may be difficult.
The QA staff must develop the knowledge locally to ensure the test evidence conforms with a jurisdiction’s typical cases.
Cost may be prohibitively expensive.
Multiple laboratories can share resources and make joint purchases; external test providers could develop materials to lower the cost.
Test must be submitted to the lab by an outside LEA.
Choosing which law enforcement agency (LEA) to work with should be decided locally based on the relationship between lab management and the LEA.
Not all LIMS are equipped to easily flag and track test cases.
Labs can choose to either use a Laboratory Information Management System (LIMS) with this functionality or develop an in-house system to flag test cases.
Labs must ensure results are not released as real cases.
The QA team will need to work with individuals in other units of the lab to prevent accidental releases. It may also be useful to have contacts in the submitting LEA or local District Attorney’s office.
Proficiency tests could impact metrics, so labs need to decide whether to include them.
These decisions must be made on a lab-by-lab basis; a consortium of labs or organizations such as AFQAM can aid in standardization.
Blind testing challenges the cultural myth of 100% accuracy.
Senior lab management must champion blind testing and show that adding it as a tool will demonstrate both the quality of examiners and help labs discover and remedy errors.

Learn More

 

Watch the HFSC webinar, “Crime Lab Proficiency and Quality Management.”

Dr. Robin Mejia discusses “Implementing Blind Proficiency Testing” in the CSAFE webinar