New Study Explores Public Beliefs About the Reliability of Forensic Science

A forensic scientist looking at prints on a computer screen.

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence

 

As with many scientific fields, forensic science has faced public and expert criticism since its conception. In response, the discipline must find ways to increase confidence in its methods and usage. One potential means of doing so was examined in a recent paper by Carlos Miguel Ibaviosa and Jason M. Chin, who posited that increased transparency and openness could solve forensic science’s public image problem.

The paper “Beyond CSI: Calibrating public beliefs about the reliability of forensic science through openness and transparency” looks at the criticisms surrounding forensic science in three stages. To begin, the authors look closely at studies that have examined the CSI Effect, which hypothesizes that procedural shows like CSI, which show forensic science to be infallible, give the public an unrealistic view of the field, which could then impact how forensic evidence is weighed in court. Most hypotheses assume this effect causes the public to view forensic science in an overly-positive and trusting way. However, after reviewing studies with these hypotheses, the paper finds that findings are inconsistent and methods are poorly constructed.

After determining that the general public is not strongly swayed by versions of forensic science they see depicted on TV, the authors review studies performed over the past 11 years that have found the public’s view of forensic science to be one of skepticism.

Although the five studies reviewed were not all performed by the same researchers, their methods were relatively comparable, and all looked at participants’ ratings of reliability for multiple different forensic tests, including DNA and bitemark evidence.

Overall, data suggested a disconnect between what experts and research regard as accurate and what the public understands as being accurate.

An example of this can be found in DNA analysis. While viewed as the gold standard within the forensic science community, two studies conducted 11 years apart showed a decrease in public trust for DNA—from a 94% reliability rating in a 2008 study to 83% in 2019.

Also miscalibrated from actual scientific findings were public views of unvalidated methods such as bitemark analysis. One study conducted in 2015 found that the public rating of the reliability for bitemark analysis was 89.26%, higher than the rating for the much more reliable fingerprint analysis, which was deemed 88.15% reliable.

The differences in language, sample size, and other sample characteristics across the studies prevent a definitive conclusion about public views of forensic science over time and their relation to expert findings on reliability. However, the authors suggest the findings still indicate a cause for concern for forensic evidence practitioners and others in the forensic science field.

Following the reviews of these studies and the lack of impression the CSI effect has on the public, the question for the authors of the paper became what, then, was responsible for the disconnect regarding reliability of forensic evidence between experts and the public?

The suggested answer is that the ability of DNA methodologies to detect previous errors, that in many cases led to wrongful convictions, was widely reported on by news coverage, as were the reports by academic bodies criticizing some of the methods responsible for such miscarriages of justice. This could have contributed to an overall public distrust in forensic science that must now be mitigated, and thus the paper turns to possible ways in which the field can bolster its credibility.

The paper’s recommendations for improved public perception and credibility focus on three components supported by research:

  1. Epistemic trust. Epistemic trust is the trust in knowledge given to us by others. This trust, on the part of the public, consists of the perceived competence of the researcher, the benevolence they show regarding improving society, and the integrity with which they follow scientific principles. Acknowledging mistakes and uncertainty in their work secures public epistemic trust in a researcher.
  2. The promotion of openness and transparency in the scientific field. When this is done, high-quality science will be distinguishable from low-quality science, as the public and scientists involved will be able to review the data and methods of different studies. Even an expressed intention of transparency has been shown to strengthen the epistemic trust of the field.
  3. Alignment with public expectations. Studies found that participants view questionable research practices, such as selective reporting, as highly morally unacceptable, despite their use not being outwardly illegal. Following these preferences will show a willingness to engage with the public as well as a dedication to good methodology.

Read the Study

Beyond CSI: Calibrating public beliefs about the reliability of forensic science through openness and transparency, Science & Justice, published online Feb. 17, 2022.

Podcast Episode Discusses Weakness in Eyewitness Identifications and Their Use in the Courtroom

Empty Courtroom

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence (CSAFE)

 

Eyewitness identification was discussed in episode seven of The Ongoing Transformation, a podcast sponsored by Arizona State University and the National Academies of Sciences, Engineering, and Medicine (NAS). Jed Rakoff, senior United States district judge for the Southern District of New York and who worked with the National Academics to publish the 2014 report on eyewitness identification, spoke about his book, “Why the Innocent Plead Guilty and the Guilty Go Free: And Other Paradoxes of Our Broken Legal System.”

Although eyewitness identification is a form of evidence very compelling to jurors, Rakoff suggested there are many reasons such evidence should be met with more skepticism. In the 375 exonerations the Innocence Project has been involved with since 1989, it was found that nearly 70% of them involved eyewitness misidentification.

Some of the reasons for these misidentifications are simple situational causes, such as lighting, an obscured view, and a tendency to focus on a weapon rather than on the details of the person handling it.

Other reasons to be cautious of eyewitness testimony are more psychological. One concern Rakoff mentions is the racial effect, in which members of the same race are more capable of distinguishing minute facial details compared to a person of a different race. Another factor at play is memory. In an example given by Rakoff, when an eyewitness begins going through a photo lineup, they may have an image of the person they saw for a few moments in rough detail in their minds. After picking a photo, the details from their sighting and the details in the photo begin to merge until the eyewitness testifying at trial months later is certain of their wrongful identification.

A solution for decreasing this high number of eyewitness misidentifications suggested by Rakoff is to educate prosecutors on the fallibility of memory and vision and identify when those flaws affect an identification. A way he suggests this could be done is to replicate a program required of all federal judges in the United States he dubbed “baby judge school,” but whose technical name is the “Phase 1 Orientation Seminar for Newly Appointed District Judges.” This program educates judges on many concepts of the legal system, from ethical concerns they’ll need to be aware of, how to organize caseloads, and how to make evidentiary decisions. Rakoff believes a similar program could teach prosecutors more about eyewitness identifications and their limitations.

Rakoff is also in favor of adopting a U.K. practice in which criminal prosecutors spend six months working as a criminal defense attorney every three years. He believes that, among other things, this can provide prosecutors with important insights on how to handle forensic evidence in cases.

Regarding forensic science reform in general, Rakoff believes the National Commission on Forensic Science, created under President Obama and whose term lapsed under President Trump, should be renewed. In its four years, the commission made 59 recommendations to the Department of Justice that could also be applied to state police and prosecutors.

Additionally, Rakoff believes that The National Institute of Forensic Sciences should be created. This institute was a suggestion made in the National Academy of Science’s 2009 report ​​Strengthening Forensic Science in the United States: A Path Forward. According to the report, this institution would consist of unbiased scientists with no connections to law enforcement or crime labs, who would review different forensic science methods and determine how each could be improved.

To end the interview, Rakoff stated that despite the flaws and need for reform he’s seen in the criminal justice system, he’s optimistic for the future.

To listen to or read the transcript from Episode 7: Shaky Science in the Courtroom, visit https://issues.org/episode-7-shaky-forensic-science-courtroom-rakoff/.

AAFS Cooperative Agreement with NIST Provides Standards Resources and Training to the Forensic Science Community

AAFS Standards Resources & Training

The American Academy of Forensic Sciences (AAFS) announced in December 2021 a cooperative agreement with the National Institute of Standards and Technology (NIST) to develop training, tools and resources to enhance implementation efforts and broaden awareness of forensic science standards among communities of interest.

According to the AAFS news release, “Training will address technical aspects of the standards as well as challenges, practical solutions and benefits of adoption. Resources, including auditing checklists for compliance monitoring and gap analysis, will also be developed, as well as factsheets, understandable to the layperson.”

AAFS said these resources would help advance the implementation of standards and guidelines listed on the Organization of Scientific Area Committees (OSAC) for Forensic Science’s Registry.

The standards training and resources can be found on the AAFS website at www.aafs.org/research-insights-featured/standards-resources-and-training. The resources are available at no cost to the public.

The webpage includes information about the cooperative agreement, upcoming webinars, videos on the standards, standards checklists (coming soon) and the AAFS Standards Factsheets.

The AAFS Standards Factsheets provide a summary of each standard and highlight its purpose, why it is important, and what its benefits are. AAFS notes that the factsheets are in continuous production, and more will come soon. There are currently 12 published factsheets available to download.

The factsheets include:

  • ANSI/ASB Standard 018 Standard for Validation of Probabilistic Genotyping Systems
  • ANSI/ASB Standard 020 Standard for Validation Studies of DNA Mixtures, and Development and Verification of a Laboratory’s Mixture Interpretation Protocol
  • ANSI/ASB Standard 036 Standard Practices for Method Validation in Forensic Toxicology
  • ANSI/ASB Standard 037 Guidelines for Opinions and Testimony in Forensic Toxicology
  • ANSI/ASB Standard 040 Standard for Forensic DNA Interpretation and Comparison Protocols
  • ANSI/ASB Standard 061 Firearms and Toolmarks 3D Measurement Systems and Measurement Quality Control
  • ASTM E2329-17 Standard Practice for Identification of Seized Drugs
  • ASTM E2548-16 Standard Guide for Sampling Seized Drugs for Qualitative and Quantitative Analysis
  • ASTM E3245-20e1 Standard Guide for Systematic Approach to the Extraction, Analysis, and Classification of Ignitable Liquids and Ignitable Liquid Residues in Fire Debris Samples
  • ASTM E3260-21 Standard Guide for forensic Examination and Comparison of Pressure Sensitive Tapes
  • NFPA-921 Guide to Fire and Explosion Investigations
  • NFPA-1033 Standard for Professional Qualifications for Fire Investigations

The Center for Statistics and Applications in Forensic Evidence (CSAFE), a NIST Center of Excellence, has several researchers who serve on the OSAC Forensic Science Standards Board (FSSB), subcommittees and resource task groups, including Jeff Salyards, a CSAFE research scientist, who serves as an FSSB member at large, and Danica Ommen, a CSAFE researcher, who serves as the chair of the Statistics Task Group. Learn more about how these groups help the development of scientifically sound standards and guidelines for the forensic science community at https://www.nist.gov/osac/osac-organizational-structure.

NIST Seeks Public Comment on Draft Report of Digital Forensic Methods

Working on a Laptop

The National Institute of Standards and Technology (NIST) has published Digital Investigation Techniques: A NIST Scientific Foundation Review. The draft report will be open for public comments through July 11, 2022.

The report reviews the methods that digital forensic experts use to analyze evidence from computers, mobile phones and other electronic devices.

According to a news release from NIST, the authors of the report examined peer-reviewed literature, documentation from software developers, test results on forensic tools, standards and best practices documents and other sources of information.

The news release also stated that the report discusses several challenges that digital forensic experts face, including the rapid pace of technological change, and recommends better methods for information-sharing among experts and a more structured approach to testing forensic tools.

NIST will host a webinar to discuss the draft report and its findings on June 1 from 1–3 p.m. EDT. For more information about the webinar and to register, visit www.nist.gov/news-events/events/2022/06/webinar-digital-investigation-techniques-nist-scientific-foundation.

Read the full news release on the report at www.nist.gov/news-events/news/2022/05/nist-publishes-review-digital-forensic-methods.

The Center for Statistics and Applications in Forensic Evidence (CSAFE), a NIST Center of Excellence, conducts research addressing the need for forensic tools and methods for digital evidence. Learn more about this research at forensicstats.org/digital-evidence.

The Innocence Project: 30 Years of Advocating for Justice Reform

Innocence Project

By Samantha Springer, a research assistant at the Center for Statistics and Applications in Forensic Evidence (CSAFE)

 

As the Innocence Project enters its 30th year, Christina Swarns, executive director of the Innocence Project, reflects in an open letter on the challenges and opportunities that lie ahead.

In the letter, In the Vanguard of Justice Reform: The Road Ahead, Swarns takes inventory of what the Innocence Project has learned throughout its three decades in the field of forensic science and reminds all of us in the discipline what the path forward will look like.

Although the Innocence Project primarily works with DNA evidence, the organization plans to extend its advocation to some cases dealing with non-DNA evidence, including research areas CSAFE specializes in. Their foundational pillars of Restoring Freedom, Transforming Systems and Advancing the Movement have relevance for not only CSAFE but for all parties interacting with the criminal justice system.

Expanding Knowledge

One of the Innocence Project’s goals is to create a comprehensive literacy program to educate all players in the criminal process on the science of forensic evidence, including judges, attorneys and jurors. The program will help legal professionals understand the foundations of scientific evidence with the goal of reducing the rate of wrongful convictions based on the misapplication of forensic science. CSAFE is currently working with the Innocence Project on this program.

Swarns writes, “We will launch an ambitious scientific literacy program to educate system actors — from public defenders, to prosecutors, to judges — about science and the limits of forensic evidence. Because too many attorneys have too little grasp of the foundations of scientific evidence, we believe that, with this program, we can and will reduce the rate of wrongful convictions based on the misapplication of forensic science. This program — which we are undertaking with leaders in the field like the Center for Statistics and Applications in Forensic Evidence (CSAFE) — will help legal professionals understand the basics of the evidence in the cases they handle.” 

The Innocence Project collaborates with CSAFE to increase and improve forensic science literacy. One of the results of this collaboration was contributing to a special issue of Significance Magazine, published in 2019 and dedicated solely to articles regarding forensic science and statistics.

Overturning Wrongful Convictions

Based on research done by the National Registry of Exonerations, 52 percent of the wrongful conviction cases handled by the Innocence Project have been due in part to misapplications of forensic evidence. Some examples of errors made include the use of unreliable evidence, misleading expert testimony and the submission of forensic sciences that have been discredited. Following the recommendations of multiple reports, including the National Academy of Sciences’ 2009 report Strengthening Forensic Science in the United States: A Path Forward, the Innocence Project will continue working with lawmakers to create legislation that allows a retrial on the basis of discredited science.

One of the problems underlying the use of unreliable forensic evidence in a criminal trial is the insufficient validation of the scientific methods being presented. To increase the validity, and therefore the quality, of analysis methods being used, more research must be done. CSAFE is one of the organizations dedicated to such reform by conducting research on promising yet under-analyzed types of forensic evidence, such as footwear impression analysis. As well as working to bolster the validity of existing forensic evidence, CSAFE also researches new avenues in an evidence type when a previous method shows itself to be unreliable, such as the former use of comparative bullet lead analysis. Current research by CSAFE now looks into firearm analysis through toolmark comparison in both bullets and cartridge cases. Through these large and well-constructed studies, organizations like CSAFE further the potential of new forensic evidence analysis and bolsters the public opinion of forensic science.

Learn more about CSAFE’s key research areas in probability and statistics for pattern and digital evidence, cross-cutting issues and training and education at https://forensicstats.org/our-research/.

GAO Releases a Second Report on Forensic Science Algorithms

From GAO Report 21-435
GAO-21-435 — Forensic Technology: Algorithms Strengthen Forensic Analysis, but Several Factors Can Affect Outcomes
GAO-21-435 — Forensic Technology: Algorithms Strengthen Forensic Analysis, but Several Factors Can Affect Outcomes

In July, the U.S. Government Accountability Office (GAO) released the report, Forensic Technology: Algorithms Strengthen Forensic Analysis, but Several Factors Can Affect Outcomes.

This is the second report in a two-part series of technology assessments responding to a request to examine the use of forensic algorithms in law enforcement. The first report, Forensic Technology: Algorithms Used in Federal Law Enforcement (GAO-20-479SP), described forensic algorithms used by federal law enforcement agencies and how they work.

In this report, GAO conducted an in-depth analysis of three types of algorithms used by federal law enforcement agencies and selected state and local law enforcement agencies: latent print, facial recognition and probabilistic genotyping. The report discusses

  1. the key performance metrics for assessing latent print, facial recognition and probabilistic genotyping algorithms;
  2. the strengths of these algorithms compared to related forensic methods;
  3. the key challenges affecting the use of these algorithms and the associated social and ethical implications; and
  4. options policymakers could consider to address these challenges.

GAO developed three policy options that could help address challenges related to law enforcement use of forensic algorithms. The policy options identify possible actions by policymakers, which may include Congress, other elected officials, federal agencies, state and local governments and industry.

In conducting this assessment, GAO interviewed federal officials, select non-federal law enforcement agencies and crime laboratories, algorithm vendors, academic researchers and nonprofit groups. It also convened an interdisciplinary meeting of 16 experts with assistance from the National Academies of Sciences, Engineering, and Medicine; and reviewed relevant literature. CSAFE co-director Karen Kafadar, professor and chair of statistics at the University of Virginia, participated in the meeting, as well as Will Guthrie, a CSAFE Research and Technology Transfer Advisory Board member. Guthrie is chief of the Statistical Engineering Division at the National Institute of Standards and Technology.

Read More:

Learn More:

CSAFE researchers are developing open-source software tools, allowing for peer-reviewed, transparent software for forensic scientists and researchers to apply to forensic evidence analysis. These automatic matching algorithms provide objective and reproducible scores as a foundation for a fair judicial process. Learn more about CSAFE’s open-source software tools.

A NEW NAME AND A NEW HORIZON: THE WILSON CENTER FOR SCIENCE AND JUSTICE

In recognition of a $5 million dollar grant from the Wilson Foundation, CSAFE partner institution Duke University renamed its Center for Science and Justice the Wilson Center for Science and Justice. Funds from the grant will expand the center’s work in three key areas: accuracy of evidence in criminal cases, the role of equity in criminal outcomes, and the mental and behavioral health treatment needs of people in the justice system. 

CSAFE’s own Co-Director, Brandon Garrett, the L. Neil Williams Jr. Professor of Law and Faculty Director of the center, aims for Duke Law School to build a long-term presence as a national leader in reforming the US criminal justice system. The Center, originally launched in September 2019, plans to encourage and instruct students and faculty on how to bolster legal and scientific data-driven research.  

“Bringing law and science together to prevent injustice has been my life’s work, since I was a young lawyer, and now, in my work with the new Wilson Center for Science and Justice at Duke,” said Professor Garrett. 

The Wilson Center champions the integration of students, faculty, and staff across multiple disciplines, gathering their expertise in law, public policy, medicine, the arts, and sciences, to create a unique lens for criminal justice research. The philosophy of interdisciplinary collaboration driving impactful research, policy, and teaching is what forms the foundation of our educational initiatives at CSAFE.

Strengthening research in the prevention of wrongful convictions, the examination of how jurors evaluate forensic evidence, and the accuracy of evidence in court are primary objectives of the Wilson Center — all of which cannot be achieved without the practice of scientifically valid methods. At its core, CSAFE is committed to the application of verified statistical and scientific techniques to ensure the accuracy of forensic analysis and interpretation and to uphold the fair administration of justice.

With more than just a strong alignment of legal and scientific ideology, CSAFE’s partnership with the Wilson Center for Science and Justice will strengthen the field of forensic science and remedy structural inequalities ingrained in the justice system.

Click here to learn more about CSAFE’s commitment to statistical and scientific data-driven research. 

Texas Forensic Science Commission Advises Implementation of OSAC Registry Standards for Crime Laboratories

In a unanimous October 2019 decision, the Texas Forensic Science Commission recommended that all crime laboratories accredited to perform forensic analysis in the State of Texas voluntarily adopt the Organization of Scientific Area Committees for Forensic Science (OSAC) standards for forensic science. The Commission is the first regulatory body in the United States to recommend the implementation of these standards.

OSAC standards found on the OSAC Registry describe best practices, explain scientific protocols and define minimum requirements for the field. Each standard aims to ensure the reliability and reproducibility of forensic analysis results.

CSAFE partner Houston Forensic Science Center has already announced the laboratory will adopt these standards. CEO and President Peter Stout states that his team continuously seeks to improve the services provided to the community, and adopting these standards is the next step in the process.

NIST created OSAC in 2014 in partnership with the Department of Justice. The organization is comprised of roughly 560 members with expertise in 25 forensic disciplines, in addition to general expertise in scientific research, measurement science, statistics, law, and policy. At this time, 12 standards are available on the OSAC Registry, with more than 200 in development.

Where to Find Answers to Your Forensic Science Questions

Do you have questions about fingerprint identification methods, applications of criminal behavior, or another forensic science-related inquiry? Head over to the International Association for Identification research library to find over 2,000 resources to uncover the answers. Compiled by the world’s oldest and largest criminal identification organization, here you will find archives, manuscripts, books periodicals and more, all related to scientific criminal investigation. Readers can even dive back into the 19th century to learn what the field of forensic science criminal investigation was like in its infancy, leading up to today’s advancements.

Find out more about IAI’s mission to educate, critique and publish methods, techniques and research in the physical forensic science disciplines on their website. Be sure to also mark your calendars for their Educational Conference on August 9-15, 2020.

Another great resource is the NIST compilation of influential and seminal papers relevant to the quantification of the weight of evidence. Find it on their website.

OSAC Public Update and Open Discussion Recordings Now Available

OSAC Logo

Did you miss The Organization of Scientific Area Committees for Forensic Science (OSAC)  Public Update & Open Discussion meeting? Videos and presentations are now available from the June 5, 2019 meeting held at NIST.

Featured presentations include updates from the five Chairs of the Scientific Area Committees (SAC), three Resource Committees, the Statistics Task Group, and the Forensic Science Standards Board (FSSB). Review the recording to learn about the latest activities for each OSAC unit, including new standards under development, identified research gaps, challenges being addressed, and priorities for the upcoming year. Audience questions and feedback are also available for viewing. Slides from the individual presenters can be found under the agenda section on the meeting website.

The OSAC works to strengthen the nation’s use of forensic science by facilitating the development of scientifically sound forensic science standards and by promoting the adoption of those standards by the forensic science community.