New Study Explores Public Beliefs About the Reliability of Forensic Science

A forensic scientist looking at prints on a computer screen.

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence

 

As with many scientific fields, forensic science has faced public and expert criticism since its conception. In response, the discipline must find ways to increase confidence in its methods and usage. One potential means of doing so was examined in a recent paper by Carlos Miguel Ibaviosa and Jason M. Chin, who posited that increased transparency and openness could solve forensic science’s public image problem.

The paper “Beyond CSI: Calibrating public beliefs about the reliability of forensic science through openness and transparency” looks at the criticisms surrounding forensic science in three stages. To begin, the authors look closely at studies that have examined the CSI Effect, which hypothesizes that procedural shows like CSI, which show forensic science to be infallible, give the public an unrealistic view of the field, which could then impact how forensic evidence is weighed in court. Most hypotheses assume this effect causes the public to view forensic science in an overly-positive and trusting way. However, after reviewing studies with these hypotheses, the paper finds that findings are inconsistent and methods are poorly constructed.

After determining that the general public is not strongly swayed by versions of forensic science they see depicted on TV, the authors review studies performed over the past 11 years that have found the public’s view of forensic science to be one of skepticism.

Although the five studies reviewed were not all performed by the same researchers, their methods were relatively comparable, and all looked at participants’ ratings of reliability for multiple different forensic tests, including DNA and bitemark evidence.

Overall, data suggested a disconnect between what experts and research regard as accurate and what the public understands as being accurate.

An example of this can be found in DNA analysis. While viewed as the gold standard within the forensic science community, two studies conducted 11 years apart showed a decrease in public trust for DNA—from a 94% reliability rating in a 2008 study to 83% in 2019.

Also miscalibrated from actual scientific findings were public views of unvalidated methods such as bitemark analysis. One study conducted in 2015 found that the public rating of the reliability for bitemark analysis was 89.26%, higher than the rating for the much more reliable fingerprint analysis, which was deemed 88.15% reliable.

The differences in language, sample size, and other sample characteristics across the studies prevent a definitive conclusion about public views of forensic science over time and their relation to expert findings on reliability. However, the authors suggest the findings still indicate a cause for concern for forensic evidence practitioners and others in the forensic science field.

Following the reviews of these studies and the lack of impression the CSI effect has on the public, the question for the authors of the paper became what, then, was responsible for the disconnect regarding reliability of forensic evidence between experts and the public?

The suggested answer is that the ability of DNA methodologies to detect previous errors, that in many cases led to wrongful convictions, was widely reported on by news coverage, as were the reports by academic bodies criticizing some of the methods responsible for such miscarriages of justice. This could have contributed to an overall public distrust in forensic science that must now be mitigated, and thus the paper turns to possible ways in which the field can bolster its credibility.

The paper’s recommendations for improved public perception and credibility focus on three components supported by research:

  1. Epistemic trust. Epistemic trust is the trust in knowledge given to us by others. This trust, on the part of the public, consists of the perceived competence of the researcher, the benevolence they show regarding improving society, and the integrity with which they follow scientific principles. Acknowledging mistakes and uncertainty in their work secures public epistemic trust in a researcher.
  2. The promotion of openness and transparency in the scientific field. When this is done, high-quality science will be distinguishable from low-quality science, as the public and scientists involved will be able to review the data and methods of different studies. Even an expressed intention of transparency has been shown to strengthen the epistemic trust of the field.
  3. Alignment with public expectations. Studies found that participants view questionable research practices, such as selective reporting, as highly morally unacceptable, despite their use not being outwardly illegal. Following these preferences will show a willingness to engage with the public as well as a dedication to good methodology.

Read the Study

Beyond CSI: Calibrating public beliefs about the reliability of forensic science through openness and transparency, Science & Justice, published online Feb. 17, 2022.

OSAC Footwear & Tire Subcommittee Develops Process Map

An overview of the Footwear and Tire Examination Process Map developed by the OSAC Footwear & Tire Subcommittee

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence (CSAFE)

 

On June 8, 2022, the Organization of Scientific Area Committees for Forensic Science’s (OSAC) Footwear & Tire Subcommittee published a current practice document for footwear and tire examination.

The 37-page document consists of multiple process maps that cover a range of practices in the field of footwear and tire examination, including casts, gel lifts, known and unknown assessments, and different types of substrates with or without the presence of blood. Additionally, the flowcharts cover administrative processes such as verification and reporting, technical assessments, and administrative assessments.

The current practice document defines its purpose as five-fold:

  • help improve efficiencies while reducing errors,
  • highlight gaps where further research or standardization would be beneficial,
  • assist with training new examiners,
  • develop specific laboratory policies, and
  • identify best practices.

The document represents current practices instead of best practices and therefore does not necessarily endorse all the methodologies shown in the multiple process maps to ensure practitioners can find the process their lab uses. According to an article published by the National Institute of Standards and Technology (NIST), David Kanaris, Chair of the OSAC subcommittee, plans to release a more interactive version of the document in the future.

NIST facilitated the development of this process map through a collaboration between the NIST Forensic Science Program and OSAC’s Footwear & Tire Subcommittee.

Other OSAC subcommittees have released their process maps for other forensic science areas, including speaker recognition, DNA, friction ridge examinations, and firearms examinations.

CSAFE researchers Alicia Carriquiry, CSAFE director, and Jacqueline Speir, an associate professor at West Virginia University, are members of the OSAC Footwear & Tire Subcommittee.

Learn more about CSAFE’s work on footwear impression analysis at https://forensicstats.org/footwear/.

Podcast Episode Discusses Weakness in Eyewitness Identifications and Their Use in the Courtroom

Empty Courtroom

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence (CSAFE)

 

Eyewitness identification was discussed in episode seven of The Ongoing Transformation, a podcast sponsored by Arizona State University and the National Academies of Sciences, Engineering, and Medicine (NAS). Jed Rakoff, senior United States district judge for the Southern District of New York and who worked with the National Academics to publish the 2014 report on eyewitness identification, spoke about his book, “Why the Innocent Plead Guilty and the Guilty Go Free: And Other Paradoxes of Our Broken Legal System.”

Although eyewitness identification is a form of evidence very compelling to jurors, Rakoff suggested there are many reasons such evidence should be met with more skepticism. In the 375 exonerations the Innocence Project has been involved with since 1989, it was found that nearly 70% of them involved eyewitness misidentification.

Some of the reasons for these misidentifications are simple situational causes, such as lighting, an obscured view, and a tendency to focus on a weapon rather than on the details of the person handling it.

Other reasons to be cautious of eyewitness testimony are more psychological. One concern Rakoff mentions is the racial effect, in which members of the same race are more capable of distinguishing minute facial details compared to a person of a different race. Another factor at play is memory. In an example given by Rakoff, when an eyewitness begins going through a photo lineup, they may have an image of the person they saw for a few moments in rough detail in their minds. After picking a photo, the details from their sighting and the details in the photo begin to merge until the eyewitness testifying at trial months later is certain of their wrongful identification.

A solution for decreasing this high number of eyewitness misidentifications suggested by Rakoff is to educate prosecutors on the fallibility of memory and vision and identify when those flaws affect an identification. A way he suggests this could be done is to replicate a program required of all federal judges in the United States he dubbed “baby judge school,” but whose technical name is the “Phase 1 Orientation Seminar for Newly Appointed District Judges.” This program educates judges on many concepts of the legal system, from ethical concerns they’ll need to be aware of, how to organize caseloads, and how to make evidentiary decisions. Rakoff believes a similar program could teach prosecutors more about eyewitness identifications and their limitations.

Rakoff is also in favor of adopting a U.K. practice in which criminal prosecutors spend six months working as a criminal defense attorney every three years. He believes that, among other things, this can provide prosecutors with important insights on how to handle forensic evidence in cases.

Regarding forensic science reform in general, Rakoff believes the National Commission on Forensic Science, created under President Obama and whose term lapsed under President Trump, should be renewed. In its four years, the commission made 59 recommendations to the Department of Justice that could also be applied to state police and prosecutors.

Additionally, Rakoff believes that The National Institute of Forensic Sciences should be created. This institute was a suggestion made in the National Academy of Science’s 2009 report ​​Strengthening Forensic Science in the United States: A Path Forward. According to the report, this institution would consist of unbiased scientists with no connections to law enforcement or crime labs, who would review different forensic science methods and determine how each could be improved.

To end the interview, Rakoff stated that despite the flaws and need for reform he’s seen in the criminal justice system, he’s optimistic for the future.

To listen to or read the transcript from Episode 7: Shaky Science in the Courtroom, visit https://issues.org/episode-7-shaky-forensic-science-courtroom-rakoff/.

AAFS Cooperative Agreement with NIST Provides Standards Resources and Training to the Forensic Science Community

AAFS Standards Resources & Training

The American Academy of Forensic Sciences (AAFS) announced in December 2021 a cooperative agreement with the National Institute of Standards and Technology (NIST) to develop training, tools and resources to enhance implementation efforts and broaden awareness of forensic science standards among communities of interest.

According to the AAFS news release, “Training will address technical aspects of the standards as well as challenges, practical solutions and benefits of adoption. Resources, including auditing checklists for compliance monitoring and gap analysis, will also be developed, as well as factsheets, understandable to the layperson.”

AAFS said these resources would help advance the implementation of standards and guidelines listed on the Organization of Scientific Area Committees (OSAC) for Forensic Science’s Registry.

The standards training and resources can be found on the AAFS website at www.aafs.org/research-insights-featured/standards-resources-and-training. The resources are available at no cost to the public.

The webpage includes information about the cooperative agreement, upcoming webinars, videos on the standards, standards checklists (coming soon) and the AAFS Standards Factsheets.

The AAFS Standards Factsheets provide a summary of each standard and highlight its purpose, why it is important, and what its benefits are. AAFS notes that the factsheets are in continuous production, and more will come soon. There are currently 12 published factsheets available to download.

The factsheets include:

  • ANSI/ASB Standard 018 Standard for Validation of Probabilistic Genotyping Systems
  • ANSI/ASB Standard 020 Standard for Validation Studies of DNA Mixtures, and Development and Verification of a Laboratory’s Mixture Interpretation Protocol
  • ANSI/ASB Standard 036 Standard Practices for Method Validation in Forensic Toxicology
  • ANSI/ASB Standard 037 Guidelines for Opinions and Testimony in Forensic Toxicology
  • ANSI/ASB Standard 040 Standard for Forensic DNA Interpretation and Comparison Protocols
  • ANSI/ASB Standard 061 Firearms and Toolmarks 3D Measurement Systems and Measurement Quality Control
  • ASTM E2329-17 Standard Practice for Identification of Seized Drugs
  • ASTM E2548-16 Standard Guide for Sampling Seized Drugs for Qualitative and Quantitative Analysis
  • ASTM E3245-20e1 Standard Guide for Systematic Approach to the Extraction, Analysis, and Classification of Ignitable Liquids and Ignitable Liquid Residues in Fire Debris Samples
  • ASTM E3260-21 Standard Guide for forensic Examination and Comparison of Pressure Sensitive Tapes
  • NFPA-921 Guide to Fire and Explosion Investigations
  • NFPA-1033 Standard for Professional Qualifications for Fire Investigations

The Center for Statistics and Applications in Forensic Evidence (CSAFE), a NIST Center of Excellence, has several researchers who serve on the OSAC Forensic Science Standards Board (FSSB), subcommittees and resource task groups, including Jeff Salyards, a CSAFE research scientist, who serves as an FSSB member at large, and Danica Ommen, a CSAFE researcher, who serves as the chair of the Statistics Task Group. Learn more about how these groups help the development of scientifically sound standards and guidelines for the forensic science community at https://www.nist.gov/osac/osac-organizational-structure.

AAFS 2022 Recap: Understanding Juror Comprehension of Forensic Testimony: Assessing Jurors’ Decision Making and Evidence Evaluation

Empty Courtroom

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence (CSAFE)

 

During the 74th annual American Academy of Forensic Sciences (AAFS) scientific conference, Cassidy Koolmees, a graduate student in legal psychology at Florida International University (FIU), and her colleagues presented a new study entitled Understanding Juror Comprehension of Forensic Testimony: Assessing Jurors’ Decision Making and Evidence Evaluation. This study came after assessing the findings in multiple studies conducted by CSAFE co-director Brandon Garrett et al., who found that a jury’s analysis of forensic testimony was not dependent on the strength of the language used during the testimony. Besides a slight decrease in conviction rates when inconclusive language was used, guilty verdicts remained stable across all conditions that used language indicating a match. This result suggests that the language used to express a match in forensic testimony has little impact on a jury, regardless of the strength of the language or credibility the expert claims.

Building on these findings, Koolmees and her colleagues examined whether jurors could distinguish low-quality testimony from high-quality testimony of forensic experts, using the language guidelines released by the Department of Justice (DOJ) in 2018 as an indicator of quality.

Study participants were put into one of six language-related conditions, where the number of violations of the DOJ language guidelines ranged from zero to five. Participants listened to a full mock trial that included the presentation of forensic evidence. Afterward, they were asked their verdict and how they would rate aspects of the testimony, including confidence in the verdict, clarity of forensic testimony, credibility of the forensic expert, and strength, quality, and usefulness of the forensic evidence.

Most of the dependent variables were found to have no statistical significance between conditions; confidence in the verdict, credibility of the expert, as well as the strength, usefulness, and clarity of the testimony were all consistent across groups.

The only statistically significant difference found between conditions was in the judgment of quality. Only when comparing the conditions of zero guideline violations and four and five violations were there any changes in guilty verdicts, signifying jurors may notice a change in the quality of forensic testimony only when the quality is severely low.

Overall, the study found that, similarly to previous studies, mock jurors are not sensitive to the quality of forensic evidence or to the differences in language used by the experts presenting said evidence. Further research by the FIU group currently being finalized includes versions of the study where jurors are made aware of the DOJ language guidelines before they are presented with expert testimony.

The researchers of the study share CSAFE’s desire for continued education for those involved in criminal trials. Suggestions put forth include simplified jury instructions and a video presentation of instructions. These proposed reforms align with the CSAFE goal of increasing education in forensic evidence for jurors, attorneys, judges, and other relevant parties.

CSAFE supports and oversees substantial contributions to training and education for a wide range of forensic science stakeholders. Explore CSAFE’s available learning opportunities and current training and education research projects at https://forensicstats.org/training-and-education/.

 

Publications referenced by Koolmees in her study:

How Jurors Evaluate Fingerprint Evidence: The Relative Importance of Match Language, Method Information, and Error Acknowledgment
Brandon Garrett and Gregory Mitchell

Mock Jurors’ Evaluation of Firearm Examiner Testimony
Brandon Garrett, Nicholas Scurich and William Crozier

AAFS 2022 Recap: An Internal Validation Study of the TopMatch 3D Scanner for Cartridge Cases

A CSAFE lab technician loads a tray of cartridge cases into the TopMatch 3D scanner.

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence (CSAFE)

 

At the 74th annual American Academy of Forensic Sciences (AAFS) scientific conference, Kayli Carrillo, a doctoral candidate at Sam Houston State University, presented a study that showed promising results for the future use of virtual microscopy in assisting forensic examiners with analyzing ballistic evidence. The study, performed by Carrillo and her colleagues at the Harris County Institute of Forensic Sciences in Houston, Texas, utilized a TopMatch VCM system identical to the microscopes used in CSAFE’s ballistics lab. CSAFE’s lab is part of the Roy J. Carver High Resolution Microscopy Facility at Iowa State University.

The internal validation study involved three stages of examination with known and unknown sourced cartridge cases analyzed by multiple examiners. The three phases introduced very few inconclusive determinations, and no matches were determined to be false positives or false negatives. These results indicate a study with a very high internal validity, which shows that the use of virtual comparison microscopy, specifically TopMatch software, can aid in forensic analysis.

Continued research will adopt a fourth step to further evaluate the inconclusive determinations made in the study by examiners. This step will compare such conclusions found when using VCM versus light comparison microscopy, alternatively known as 2D microscopy. Based on the promising findings of this study, the Harris County Institute of Forensic Sciences plans to utilize TopMatch microscopy in the analysis of their cartridge cases.

CSAFE researchers have made great strides in developing statistical and scientific foundations for assessing and matching firearms and toolmarks. Learn more at https://forensicstats.org/firearms-and-toolmark-analysis/.

NIST Releases Results from a Black Box Study for Digital Forensic Examiners

NIST Black Box Study for Digital Forensic Examiners

The National Institute of Standards and Technology (NIST) has published the results from a black box study for digital forensic examiners. The study, released in February 2022, describes the methodology used in the study and summarizes the results.

The study was conducted online and open to anyone in the public or private sectors working in the digital forensics field. Survey participants examined and reported on the simulated digital evidence from casework-like scenarios. NIST said study’s goal was to assess the performance of the digital forensic community as a whole.

Results from a Black-Box Study for Digital Forensic Examiners (NISTIR 8412) can be viewed at https://nvlpubs.nist.gov/nistpubs/ir/2022/NIST.IR.8412.pdf.

From Results from a Black-Box Study for Digital Forensic Examiners, page 33:

Summary Key Takeaways

Despite the limitations of the study, two key takeaways about the state of the digital evidence discipline emerged:

  • Digital forensics examiners showed that they can answer difficult questions related to the analysis of mobile phones and personal computers. Questions ranged from basic, such as identifying who the user of the phone had contacted, to advanced questions that related to the use of the TOR browser.
  • The response to the study underscored the size, variety, and complexity of the field. The study received responses from examiners working in international, federal, state, local government, and private labs whose major work included law enforcement, defense, intelligence, and incident response/computer security. There were also responses from people outside of these areas.

 

Results Available from OSAC Registry Implementation Survey

OSAC Registry Implementation Survey: 2021 Report

The Organization of Scientific Area Committees for Forensic Science (OSAC) released the results from its first annual Registry Implementation Survey. The report, published in February 2022, provides a detailed look at the respondents and the implementation status of the 46 standards represented in the survey.

In the summer of 2021, OSAC released the survey targeted at forensic science service providers from across the country. It was designed to help OSAC better understand how the standards on the OSAC registry are being used, the challenges around standards implementation and what support is needed to improve it.

The OSAC Registry Implementation Survey: 2021 Report is available at https://www.nist.gov/osac/osac-registry-implementation-survey.

From page 10 of the OSAC Registry Implementation Survey: 2021 Report:

Priority for Implementing Standards
When asked what priority survey participants considered standards implementation for their organization, half of the respondents (50%) said it was a medium priority, or important. This was followed by 34% of respondents indicating that implementation was a high priority, or very important. Twenty-three respondents (14.8%) indicated that implementation was a low priority or not a priority at this time (Figure 4).

Figure 4. Priorities for Standards Implementation
Click on image to enlarge. Figure 4. Priorities for Standards Implementation

A Study on Improving Forensic Decision Making will be the Topic of CSAFE’s February Webinar

Figure 2 from the study shows sources of cognitive bias in sampling, observations, testing strategies, analysis and/or conclusions, that impact even experts. These sources of bias are organized in a taxonomy of three categories: case-specific sources (Category A), individual-specific sources (Category B) and sources that relate to human nature (Category C).

A new study that proposes a broad and versatile approach to strengthening expert decision making will be the focus of an upcoming Center for Statistics and Applications in Forensic Evidence (CSAFE) webinar.

The webinar, Improving Forensic Decision Making: A Human-Cognitive Perspective, will be held Thursday, Feb. 17 from 12–1 p.m. CST. It is free and open to the public.

Itiel Dror
Itiel Dror

During the webinar, Itiel Dror, a cognitive neuroscience researcher from the University College London, will discuss his journal article, Linear Sequential Unmasking–Expanded (LSU-E): A general approach for improving decision making as well as minimizing noise and bias. The article was published in Forensic Science International: Synergy and co-authored by Jeff Kukucka, associate professor of psychology at Towson University.

In the article, the authors introduce Linear Sequential Unmasking (LSU-E), an approach that can be applied to all forensic decisions, and also reduces noise and improves decisions “by cognitively optimizing the sequence of information in a way that maximizes information utility and thereby produces better and more reliable decisions.”

From the Abstract:

In this paper, we draw upon classic cognitive and psychological research on factors that influence and underpin expert decision making to propose a broad and versatile approach to strengthening expert decision making. Experts from all domains should first form an initial impression based solely on the raw data/evidence, devoid of any reference material or context, even if relevant. Only thereafter can they consider what other information they should receive and in what order based on its objectivity, relevance, and biasing power. It is furthermore essential to transparently document the impact and role of the various pieces of information on the decision making process. As a result of using LSU-E, decisions will not only be more transparent and less noisy, but it will also make sure that the contributions of different pieces of information are justified by, and proportional to, their strength.

To register for the February webinar, visit https://forensicstats.org/events/.

The CSAFE Spring 2022 Webinar Series is sponsored by the National Institute of Standards and Technology (NIST) through cooperative agreement 70NANB20H019.

ASCLD Forensic Research Committee Provides Useful Resources for Researchers and Practitioners

ASCLD Forensic Research Committee

The American Society of Crime Laboratory Directors (ASCLD) Forensic Research Committee (FRC) works to identify the research, development, technology and evaluation needs and priorities for the forensic science community. The FRC has several initiatives and resources available on its website to aid both researchers and practitioners. Below, we highlight a few of those resources.

For more information, visit the FRC website: https://www.ascld.org/forensic-research-committee/.

Collaboration Hub

The FRC collaboration hub hosts the Researcher-Practitioner Collaboration Directory. The directory helps connect researchers with ongoing projects to practitioners who are willing to participate in the studies. The searchable directory includes descriptions of each project, including an abstract and the estimated participant time involved. Researchers can easily submit their projects for inclusion in the directory by completing an online form.

ASCLD Research Priorities

FRC has created a list of high-impact research opportunities that help to identify key areas where impactful research would support the forensic science community and enhance lab operations. The research priorities list for 2022-2024 can be downloaded at https://www.ascld.org/wp-content/uploads/2021/12/ASCLD-Research-Priority-Areas-2022-2024.pdf.

Lightning Talks

The FRC hosts a virtual “Lightning Talks” series to highlight new and emerging research in all areas of forensic science. Each episode features three short talks given by practitioners, researchers or students. Previous Lightning Talks are archived on FRC’s YouTube page.

Laboratories and Educators Alliance Program (LEAP)

LEAP facilitates collaborative research between academia and forensic science laboratories. This program identifies forensic science needs and provides a platform for laboratories, researchers and students to seek projects aligning with their mutual research capabilities. The FRC website includes a map of LEAP partners, a short video explaining LEAP and sign-up forms for crime labs and universities. LEAP is a joint effort between ASCLD and the Council of Forensic Science Educators (COFSE).

Validation and Evaluation Repository

The Validation and Evaluation Repository is a list of unique validations and evaluations conducted by forensic labs and universities. ASCLD’s summary of the repository states, “It is ASCLD’s hope that this listing will foster communication and reduce unnecessary repetition of validations and evaluations to benefit the forensic community.” The searchable repository is available at https://www.ascld.org/validation-evaluation-repository/

Research Executive Summaries

The Future Forensics Subcommittee of the FRC has initiated the publication of brief executive summaries of the recent literature within the forensic sciences. The summaries are written by ASCLD members and are meant to provide a brief overview of noteworthy publications and trends in the literature. Currently, the summaries include reviews in the areas of fingermarks, controlled substances, paint and glass evidence, forensic toxicology, forensic biology, gunshot residue analysis and firearms and toolmarks.