Category: Statistical Foundations

When it comes to a piece of evidence, a jury wants to know just how likely it is that a crime scene bullet or shoeprint matches a given suspect. An analysis may almost certainly be correct, but how does an examiner account for the remote possibility that it wasn’t the same gun?

Quantifying the weight of evidence is no simple task, and integrating room for error and uncertainty has historically proven difficult in forensic science.

NIST takes a closer look at how strategies have evolved and shared the newest research in a publicly available collection of influential and seminal papers. The curated content is a valuable resource for anyone interested in learning more about the critical role quantifying evidence plays in the pursuit of justice.

The best way to approach the weight of evidence is often highly debated amongst the forensic science community and researchers. A lively discussion on limitations and implementation of statistical methods for quantifying the weight of evidence can be found by reviewing the 2016 NIST Technical Colloquium: Quantifying the Weight of Forensic Evidence session.

One approach to quantifying evidence, the likelihood ratio, is examined in detail by CSAFE team members and NIST personnel in a 2018 CSAFE webinar. View the panel discussion in the CSAFE education center.

Interested in training on this issue by our expert statisticians? Please contact our team.

Just how closely related are methods for forensic toolmark and firearm examination? Though seemingly quite similar, analysis techniques for one may not be transferrable. CSAFE researchers take a closer look at this question in a new 2019 paper published in the Journal of Forensic Science.

In “Adapting the Chumbley Score to Match Striae on Land Engraved Areas (LEAs) of Bullets,” investigators examine if a statistical method developed for screwdriver markings successfully identifies striations on bullets.

While exploring Chumbley method performance on bullet analysis as opposed to screwdrivers, CSAFE researchers compare error rates. Results show that when judged against toolmarks, error rates increase.

Researchers discovered that parameter changes strongly impact the performance of the test. Next up, the CSAFE team plans to increase the test’s power with bullet-to-bullet comparisons.

Review the full article to learn more about this study. If you’d like a quick snapshot, view the CSAFE poster from AFTE 2018. For more information on CSAFE firearms and toolmark projects, visit our research page.

Just because a technique has been used for over a 100 years, does that make it scientifically sound or reliable? The forensic science community is taking a closer look at this question in regards to fingerprinting, a long used tool in criminal investigations to help convict or rule out suspects.

News website Vox.com recently released a new video “How Reliable is Fingerprint Evidence,” featuring a forensic science expert that walks viewers through how fingerprinting is actually done, and digs a little deeper into the concerns surrounding its use.

CSAFE researcher and Professor of Criminology, Law and Society from University of California, Irvine Simon Cole also weighs in. “(Fingerprinting) is a useful tool that obviously has value, but I think it’s problematic to overstate its value especially in a criminal justice context.”

The video highlights the famous case of Brandon Mayfield and the 2004 Madrid train bombing where a false positive fingerprint identification had major consequences. Fingerprinting requires that investigators visually compare prints for similarity and differences. The Mayfield case clearly shows that due to the reliance on human judgment, error and cognitive bias can affect accuracy.

“I would suggest learning from history and realizing none of these things are going to be error free. So there’s going to be mistakes and errors and screw-ups but also evidence is inherently probabilistic,” Cole recommends.

In the search for more reliable technology, Cole and the team at CSAFE are working to use statistics to address key forensic science challenges in an effort to reduce human error and prevent the conviction of innocent people. Learn more about our research on the CSAFE website. To partner with CSAFE in developing increased scientific foundations for forensic evidence analysis, contact us to discuss collaboration opportunities.

This post is based on the article “Statistical Issues in Forensic Science” by CSAFE Co-Director Hal Stern of UCI. The full article was published in March 2017.

The way that forensic evidence and other expert testimony is presented in court is determined by legal precedent, especially the 1993 U.S. Supreme Court decision in Daubert v Merrell Dow Pharmaceuticals, and Rule 702 of the Federal Rules of Evidence.  These sources identify the trial judge as the “gatekeeper” to identify which testimony to allow, and set the expectation that such testimony “is based on sufficient facts or data” and “is the product of reliable principles and methods”. A 2009 National Research Council (NRC) report and a 2016 report of the President’s Council of Advisors on Science and Technology (PCAST) question whether a number of forensic science disciplines satisfy these expectations. Each calls for additional research to support the science underlying the analysis of forensic evidence.

A multidisciplinary approach is required to carry out such research because each discipline requires technical expertise to collect and prepare evidence and additional expertise to analyze and interpret the resulting data. Statistics has a crucial role to play in helping to address the challenges of forensic science.  Examples of the way that statisticians and the field of statistics can contribute are described below.

Reliability and validity

Perhaps the most important challenge facing forensic science after the 2009 NRC and 2016 PCAST reports is the need for data that assesses the reliability and validity of forensic examinations and conclusions.  Reliability refers to whether forensic measurements or judgements would be obtained consistently.  This includes considerations of repeatability, in which one asks if the same forensic examiner would draw the same conclusion if presented with the same evidence at a different time, and considerations of reproducibility, in which one asks if the same conclusion or measurement would be obtained by other examiners. Reliability is an important first consideration but even if forensic examinations in a particular domain are reliable, that does not indicate whether they are valid or accurate.  If a fingerprint examiner concludes that a latent print at the crime scene comes from the same source as a test impression made by the suspect (this conclusion is sometimes known as an identification), we need to know how accurate that conclusion is to make an informed judgment about the weight of evidence. The PCAST report carefully discussed what should be expected in validation studies (e.g., the true status of the samples should be known, the samples should be representative of casework, etc.).

PCAST indicated that statistical methods are well developed and have been validated for single source DNA evidence (or simple mixtures) and for latent print analysis, but reported that additional studies are needed regarding the validity of forensic examinations  in other forensic science disciplines. Assessing the reliability and validity of forensic examinations is a key step in developing sound science to measure the strength of evidence and statisticians can contribute to the design and analysis of such studies.

Task-relevant information and cognitive biases

Scientists are humans, not robots. As such, the field of forensic science, as any other domain involving human judgements, has the potential to be compromised by cognitivebias and other cognitive factors which can influence how data is collected, analyzed and findings are communicated. Cognitive bias is a term meant that refers to disconnects between observed behavior, and what would be considered by most as rational decision making.

There are a number of well-known cognitive biases that have been identified in other areas of human decision-making.  One example is known as confirmation bias, where examiners may tend to favor interpretations that confirm their own preconceptions about a suspect or the evidence. Another example are framing effects, where a forensic examiner may evaluate evidence differently depending on what  contextual information about the suspect or the crime scene was provided.

Statisticians can play a role in teams that study cognitive bias and in teams that try to determine what information should be deemed relevant for a particular task.

Causal inference

Forensic examiners working in crime scene investigation, arson and blood spatter analysis attempt to reconstruct the events leading up to a crime based on evidence found at the crime scene. This process is an effort to infer the causes of observed effects. These examiners face a challenging task because it is difficult to carry out realistic controlled experiments that would allow one to reliably distinguish between competing explanations. For example, the problem of determining if a fire developed naturally or by the use of an accelerant.

Statistical collaboration with practitioners in relevant disciplines will be valuable in strengthening inferences in these settings.

Case processing and procedures

Any process can benefit from careful analysis to understand potential limitations, bottlenecks or sources of error, and the forensic evidence analysis process is no exception.

Crime laboratories are often faced with an overwhelming workload, leading to backlogs that can slow the treatment of evidence. At the same time, maintaining quality forensic examinations  requires a quality assurance program that incorporates reanalysis of evidence or verification of conclusions.

Statistical methods can play a role in designing quality assurance programs that can improve efficiency of lab operations while simultaneously insuring the accuracy of conclusions.

Testifying on forensic evidence

Appropriate ways for presenting forensic evidence analysis in the courtroom is an important area of research. Several studies demonstrate the difficulty that jurors can have in understanding statistical ideas like the likelihood ratio and Bayes factor. The forensic science research community is still examining the best way to presnt such evidence in the courtroom.

Statisticians have an important role to play in developing approaches to presenting quantitative evaluations of evidence and in the design and analysis of juror studies to assess the effectiveness of alternative approaches.

How CSAFE Research is Advancing the Field of Forensics

Statisticians at the Center for Statistics and Applications in Forensic Evidence (CSAFE) work in partnership with NIST researchers, forensic science practitioners and scientists in a variety of disciplines to address the issues raised in the NRC and PCAST reports.

Our team is contributing to strengthening the statistical foundations of pattern and digital evidence through the study of existing methods and the development of new statistical to analyze and interpret data and evidence. Our goal is to help forensic scientists in their pursuit of reliable and accurate analyses of forensic evidence.

Learn more about our research.

This post is based on the article “Statistical Issues in Forensic Science” by CSAFE Co-Director Hal Stern of UCI. The full article was published in March 2017.

This post is based on the article “Statistical Issues in Forensic Science” by CSAFE Co-Director and Professor of Statistics Hal Stern of Univeristy of California, Irive. Review the full article in the Annual Review of Statistics and Its Application.

When it comes to solving crimes, forensic examiners play a key role in the investigation process and trial proceedings. Their responsibilities vary widely depending on the type of crime and available evidence.

For example, examiners may be called on to identify time of death in a homicide case, reconstruct the events of crime from blood spatter patterns, or take a close look at physical evidence such as broken glass or fingerprints found at the scene. Their analysis of evidence provides crucial details that help in the search for the culprit. The remainder of the discussion here is focused on the role of forensic examiners in attempting to identify if a suspect is linked to particular item of evidence in the case.

It is important to understand that the goal in examining a single piece of evidence is actually not to determine the guilt or innocence of the suspect. Rather, it is an effort to identify if the evidence is associated with the suspect, e.g., if the suspect is a possible source of a fingerprint found at a crime scene. Any number of legitimate reasons could lead to finding evidence from a suspect at the crime scene, such as a visit to the scene at a different time.

Exploring the Different Types of Evidence

• DNA – DNA found in biological samples recovered at the crime scene is among today’s most powerful forms of forensic evidence. The forensic examiner’s task when analyzing this type of evidence is to determine if the DNA profile of a suspect matches the DNA profile found in the crime scene sample. They then must assess the significance of this agreement.
• Trace Evidence – Refers to evidence types characterized as a fragment or sample of a larger object that is left behind at the time of the crime. This type of evidence could include glass fragments from a broken window, hairs from an individual, or fibers from clothing or carpet. The challenge here is to determine if a sample of trace evidence from the crime scene matches another sample obtained from a suspect (or perhaps from an object in the suspect’s possession).
• Pattern evidence – Evidence left at the crime scene that is the result of an impression left by a person or object is known as pattern evidence. The forensic examiner must attempt to determine if the impression found at the crime scene matches the pattern of an analogous sample obtained from the suspect or from an object known to belong to the suspect. Types of pattern evidence include fingerprints, shoeprints, documents/handwriting, toolmarks and firearm impressions.
• Digital evidence – Digital technology is playing a larger and larger role in criminal and civil investigations. Digital evidence can refer to any information obtained from a device implicated in an investigation. Examples include images or messages found on a smartphone belonging to a suspect. Digital evidence can be challenging to analyze due to the wide variety of different evidence types that may be found on digital devices.

Recent Events Have Raised Concerns About Current Methods

Forensic examiners summarize their evidence analysis in reports that play a crucial role in investigations and trials. Recent events have called into question the scientific and statistical foundations of evidence analysis and interpretation.

This includes the release of two reports, the 2009 National Research Council (NRC) report and a 2016 report of the President’s Council of Advisors on Science and Technology (PCAST), in which committees comprised of scientists from a number of disciplines questioned whether there was sufficient quantitative data supporting the statements made by forensic examiners in a number of disciplines.

Beyond these reports, an additional source of concern is a number of cases in which forensic science errors have been identified. A famous example is the very public case involving American lawyer Brandon Mayfield. Fingerprint examiners from the Federal Bureau of Investigation (FBI) mistakenly identified Mayfield as the source of a latent fingerprint found at the scene of a 2004 train bombing in Spain.

The Innocence Project, a nonprofit legal organization founded in 1992, details countless other situations where unreliable or improper forensic science led to wrongful convictions. The organization has been instrumental in freeing more than 300 wrongfully convicted individuals through the beginning of 2018 and improper forensic science is identified as a contributor in roughly half of these cases.

Why We Need Statistics In Forensic Evidence Analysis

Both the 2009 NRC report and the 2016 PCAST report emphasized the need for additional study of forensic science methods. It is clear that statistical methods have a key role to play in strengthening the scientific foundations of forensic examinations. Statistics is the science concerned with designing studies and experiments, analyzing and interpreting the results, and summarizing the information obtained. As such, it can contribute to studies aimed at determining the accuracy of the conclusions drawn by forensic examiners, addressing cognitive biases, examining the influence of irrelevant information on analysis, evaluating modifications to case processing procedures and more.

This post is based on the article “Statistical Issues in Forensic Science” by CSAFE Co-Director Hal Stern of UCI published in Annual Review of Statistics and Its Application.

Imagine a world without standard measurements.  How would we quantify the weight of our food, or the length of material for clothes?  Without a standard system, confusion abounds. In 1901 the U.S. government created the National Institute of Standards and Technology (NIST). NIST is a non-regulatory federal agency within the U.S. Department of Justice.

Building the Foundation of Innovation

NIST’s goal is to promote U.S. innovation and industrial competiveness through advances in measurement in science, standards and technology.  Innovation relies on the ability to observe and measure which allows product control and standard procedure implementation.

In addition to applications in industry and manufacturing, universally accepted definitions of measurement play a key role in scientific advancement. NIST researchers are at the forefront of developing new technology and commercialization.

“From the smart electronic power grid and electronic health records to atomic clocks, innumerable products and services rely in some way on technology, measurement and standards provided by the National Institute of Standards and Technology.”

Benefits of Standard Measurement in Forensic Science

Did you know that standard measurement is key in forensic science? Universal procedures for collection, analysis and interpretation of forensic evidence help protect the innocent and convict the guilty.  NIST focuses on three components of improving forensic science standards.

• Science- Leads the way in innovative research in many forensic science disciplines to include DNA, fingerprints, digital evidence and more. Experts increase forensic laboratory analytical method validity through physical reference standards and data.
• Policy- Co-chaired the National Commission on Forensic Science to raise awareness on standards such as accreditation requirements for forensic science service providers.
• Practice- Directs Organization of Scientific Area Committees for Forensic Science to develop science based standards and guidelines for forensic science disciplines.

CSAFE and NIST- Working together to Transform Forensic Science

NIST is committed to advancing efforts to solve deficiencies in forensic science standards.  In 2015, NIST established CSAFE, a Center for Excellence in forensic science.  NIST continues to provide funding and strategical guidance to CSAFE as the center works to build the scientific foundations of forensic science.  The CSAFE team benefits greatly from the expertise NIST brings to CSAFE research and education initiatives.  Working in close partnership, NIST and CSAFE are developing solutions to better recognize, collect, analyze and interpret evidence.

Sources:

https://www.nist.gov/speech-testimony/importance-basic-research-united-states-competitiveness

https://www.nist.gov/about-nist

https://www.nist.gov/topics/forensic-science