This CSAFE Center Wide Webinar was presented on March 5, 2018 by Lesley Hammer, (MA, MSc) forensic consultant at Hammer Forensics.
Presentation description: This presentation will provide an overview of footwear impression examination, explore directions that research in this discipline has been trending and where it may lead, and summarize challenges that are influencing and opportunities that are ahead for the discipline of forensic footwear evidence in the U.S.
This CSAFE Center Wide Webinar was presented on March 1, 2018 by a panel of speakers who examine the impact of the likelihood ratio in a legal context, how to communicate the likelihood ratio to lay audiences and more, followed by a time for questions from the audience.
Panelists:
Dr. Steve Lund and Dr. Hari Iyer- NIST Mathematical Statisticians
Dr. Hal Stern- CSAFE Co-Director and Professor of Statistics, University of California, Irvine
Dr. William Thompson- CSAFE Researcher and Professor Emeritus at University of California, Irvine
David Kaye- Distinguished Professor of Law, Pennsylvania State Law School
The Center for Statistics and Applications in Forensic Evidence Director and Distinguished Professor of Statistics Dr. Alicia Carriquiry walks us through a landmark case that got statistics wrong.
When forensic practitioners, lawyers and other expert witnesses use statistics appropriately, juries have a tool that can help them make a more informed decision about the guilt or innocence of a suspect. However, misrepresenting statistics in the courtroom can come at a high cost.
The Sally Clark Case is an infamous criminal case from the United Kingdom that illustrates how the use of inaccurate statistics in forensic science can lead to grave injustices.
In December 1996, Sally Clark was home alone with her apparently healthy baby Christopher, aged 2 and half months. Sometime during the evening, Sally found her baby unresponsive and called an ambulance. Resuscitation attempts failed and the baby was pronounced dead. A post-mortem report suggested death of natural causes and a possible lower tract respiratory infection.
Less than two years later in January of 1998, Sally’s second baby Harry died at age two months under almost identical circumstances. The post-mortem report described signs of recent bleeding at the back of the eyes and in the spinal cord. Harry was an apparently healthy baby who was carefully observed by the UK’s Care of Next Infants program, which provides resources to families who have previously experienced the sudden death of an infant.
Alerted by the same pathologist who had performed the autopsy on Christopher, the police arrested Sally and her husband on murder charges. Officials quickly dropped the charges against Steve Clark, but accused Sally of smothering her two children.
Aside from the two dead infants, there was no other evidence to suggest murder. Sally’s friends and family praised her character and her mothering. Much of the evidence presented at trial for the prosecution came from medical experts, including renowned pediatrician Sir Roy Meadows.
The statistical argument
Using epidemiological data from the UK, Sir Meadows argued that two unexplained infant deaths (SIDS deaths) are extremely rare in a family such as the Clarks.
In the 1990s, the incidence of SIDS in the UK was approximately 1 in 8,500 in middle-class families with no known risk factors. Sir Meadows further argued that the two deaths could be considered to be independent events and that the probability of observing two SIDS deaths in the Clark family could be computed as 1/8,500 squared, or 1 in 75 million.
The question is, can these two deaths really be treated as independent? Two SIDS deaths occurred in the same home under the supervision of the same two parents. How likely is it that those are unrelated events?
Jury confusion: The Prosecutor’s Fallacy
It’s not hard to see that these statistics could easily be misunderstood and are likely prejudicial. Many people believe that the chance of a rare event happening is the same as the chance of a suspect’s innocence, an error known as the Prosecutor’s Fallacy. In reality, these two probabilities are not equal: saying that there is a 1 in 73 million chance that the babies died of SIDS is not the same as saying that there is a 1 in 73 million chance that the mother did not kill them.
Understanding the Correct Statistical Applications
In Sally’s case, we have two extremely rare events to examine. Either the two babies died of SIDS, or the two babies were murdered. The probabilities of both events are difficult, if not impossible, to compute precisely, and both are likely to be extremely small. Luckily, we do not need precise estimates for those probabilities. Instead, we just need a reasonable estimate of their ratio.
Using UK crime statistics and the exact same reasoning that Sir Meadows used in his testimony against Sally, we find that the probability that two infants will be murdered in the same household is just 1 in 2 billion.
When considering the likelihood of two SIDS deaths and the likelihood of two murders, the odds of Sally’s guilt are very small, about 4%. It is in fact much more likely (about 95% more likely) that the babies died of SIDS than that they were murdered by their mother.
What happened to Sally?
In 2003 Sally’s conviction was overturned on the basis of additional medical evidence of a possible bacterial infection in Harry. Sir Meadows used the same statistical arguments in other similar cases and was found guilty of professional misconduct in 2005, though the verdict was later overturned.
Sally Clark died on March 15, 2007, from a suspected heart attack thought to be a result of alcohol abuse. She was 42 and spent four years in prison because neither the judge nor the jurors could correctly interpret the significance of the numbers put forth by the expert witnesses.
The Sally Clark case urges caution when using statistics in the courtroom. All parties must remember that there are always two (or more) sides to the story. Forensic scientists, jurors, lawyers and judges need to consider the probability of the evidence under the two competing hypotheses: the suspect is guilty or the suspect is not guilty. The ratio of these two probabilities is the likelihood ratio, and it can be approximated, if not explicitly calculated, in many cases. High values of the likelihood ratio would tend to support the hypothesis of guilt, while low values would tend to support the hypothesis of not guilty.
Armed with an accurate understanding of this key statistical concept, courts can prevent wrongful convictions such as Clark’s through the correct interpretation of evidence.
This CSAFE Center Wide Webinar was presented by Xiao Hui Tai from Carnegie Mellon University on January 31, 2018.
Description:
We will describe work that we have done on an automated method to compare breechface marks on 2D images of cartridge cases, and how this transfers to 3D topographies. We will discuss various studies and compare the results for 2D and 3D data. In the future, such algorithms may be used by examiners for blind verification or for quantifying the weight of evidence.
Is the field of science-based solely on logic and facts? A closer look reveals that science is not always as systematic and foolproof as we may think.
Scientists are humans, not robots. As such, the field of science has the potential to be compromised by human error, bias and other cognitive factors.
CSAFE researchers and psychologists Dr. Daniel Murrie and Dr. Sharon Kelley from the University of Virginia describe how human factors can influence the accuracy of scientific work as a whole, and the specific impact they can have on forensic science.
Defining Human Factors
Dr. Murrie explains that human factors are relevant any time that the human mind or behavior is part of the scientific process. “Human factors, including cognitive bias, can influence how we collect and analyze data, or how we communicate the findings,” he said.
The human brain is powerful and typically steers us in the right direction.
“Our minds are very efficient and that usually serves us well,” Murrie said. “Our brains develop many shortcuts or heuristics that help us make reasonably good decisions fast.”
While this approach to decision making seems practical and beneficial, it has a downside.
“There are a few times when that cognitive infrastructure can lead us to take short cuts or jump to conclusions,” Murrie said. “This can leave us vulnerable to bias in ways that aren’t helpful to science, accuracy or objectivity.”
Human Factors in Forensic Evidence Analysis
The 2009 NAS report calling for reform in forensic science has led to more awareness of human factors and driven new research on cognitive bias in evidence analysis.
For instance, when a fingerprint or shoeprint examiner compares the print of a suspect to a print from the crime scene, there is potential for human factors to bias the examiner’s decision on a match.
Dr. Kelley explains how.
“General social and cognitive psychology research indicates that (A) humans often see what they expect to see and that (B) humans tend to seek out and interpret information in a way that supports their pre-existing beliefs,” she said.
For example, forensic science examiner may have had an opinion about the guilt or innocence of a suspect before even beginning the examination. Whether intentional or not, it could lead to biased results.
Dr. Kelley emphasizes that forensic scientists must be aware of potential bias before expressing opinions or decisions on the outcome of evidence analysis.
Why Is Awareness of Bias, not Enough?
“The good news is that awareness of biases in forensic science is probably at an all-time high. The bad news is that people don’t quite know what to do with this new awareness,” Murrie said.
One concern from recent research is the “bias blind spot,” as cognitive psychologists call it.
“People generally recognize that bias is a problem, but they only see it in others—not themselves,” Murrie said.
Kelley explains that for those who do recognize their own biases, it’s hard to change on your own.
“Introspecting and knowing about the bias and then just trying hard not to be biased is not enough,” Kelley said. “We know from research that these biases often aren’t conscious and you can’t just scan yourself and check for them. Biases still creep into our decision making.”
Instead of squinting your eyes and focusing hard not to be biased, Kelley says procedural and systematic changes are needed.
Strategies to Lessen the Impact of Human Factors in Crime Labs
According to Kelley, crime labs are a very heterogeneous group. They differ in their understanding of cognitive bias and strategies to combat the issue.
So how can researchers like Murrie and Kelley help crime labs reduce the impact of human factors in their investigations?
Two strategies CSAFE is working on to reduce bias are blind verification procedures and context management.
Blind verification procedures serve as checks and balances when analyzing evidence. In this method, a second examiner reviews a case with no information about what the first examiner concluded. Crime laboratories then have two independent decisions to compare. When the two examiners agree, there is more confidence that the analysis is accurate.
Context management involves limiting unnecessary contextual information about the suspect or the crime scene that is irrelevant to the evidence analysis task. For example, when comparing a set of fingerprints, the examiner doesn’t need to know the race or criminal record of the suspect, or even the results of DNA analyses, to do their job. Reducing potentially biasing information increases objectivity in evidence analysis.
Read more about how CSAFE research is addressing context management in crime laboratories.
Recent advancements in technology have brought forensic science to the forefront in criminal investigations. While investigators are increasingly relying on the scientific foundations of forensic science, constitutional regulation may not be keeping up.
Historically, the U.S. Supreme Court has not taken a firm stance on the type of oversight the constitution should provide concerning forensic science. However, increased reliance on evidence and analysis techniques in the courtroom is persuading the government to take a closer look.
CSAFE researcher and University of Virginia White Burkett Miller Professor of Law and Public Affairs and Justice Thurgood Marshall Distinguished Professor of Law Brandon Garrett recently released a comprehensive review of the latest issues on constitutional regulation of forensic science.
From the article:
“Despite decades of missed opportunities to adequately regulate forensics, in recent rulings the Supreme Court and lower courts increasingly focus on sound litigation of forensics. In an era of plea bargaining, the accuracy of forensic analysis depends far less on cross-examination at trial, and far more on sound lab techniques, full disclosure of strengths and limitations of forensic evidence to prosecutors and the defense, and careful litigation.”
Learn more in the Washington and Lee Law Review journal.
Imagine a world without standard measurements. How would we quantify the weight of our food, or the length of material for clothes? Without a standard system, confusion abounds. In 1901 the U.S. government created the National Institute of Standards and Technology (NIST). NIST is a non-regulatory federal agency within the U.S. Department of Justice.
Building the Foundation of Innovation
NIST’s goal is to promote U.S. innovation and industrial competiveness through advances in measurement in science, standards and technology. Innovation relies on the ability to observe and measure which allows product control and standard procedure implementation.
In addition to applications in industry and manufacturing, universally accepted definitions of measurement play a key role in scientific advancement. NIST researchers are at the forefront of developing new technology and commercialization.
“From the smart electronic power grid and electronic health records to atomic clocks, innumerable products and services rely in some way on technology, measurement and standards provided by the National Institute of Standards and Technology.”
Benefits of Standard Measurement in Forensic Science
Did you know that standard measurement is key in forensic science? Universal procedures for collection, analysis and interpretation of forensic evidence help protect the innocent and convict the guilty. NIST focuses on three components of improving forensic science standards.
CSAFE and NIST- Working together to Transform Forensic Science
NIST is committed to advancing efforts to solve deficiencies in forensic science standards. In 2015, NIST established CSAFE, a Center for Excellence in forensic science. NIST continues to provide funding and strategical guidance to CSAFE as the center works to build the scientific foundations of forensic science. The CSAFE team benefits greatly from the expertise NIST brings to CSAFE research and education initiatives. Working in close partnership, NIST and CSAFE are developing solutions to better recognize, collect, analyze and interpret evidence.
Sources:
https://www.nist.gov/speech-testimony/importance-basic-research-united-states-competitiveness
CBS launched the first episode of CSI: Crime Scene Investigation in the year 2000, and suddenly forensic science was the latest craze. What was once a foreign field to many Americans was now at the forefront of our nation’s curiosity. Nearly 20 years later, viewers are still on the edge of their seats, captivated by stories of crime.
Two important questions to consider:
While it may come as a surprise to the everyday citizen, the forensic science community wants you to know that the answers are Hollywood is usually wrong, and television influences public perception more than you might think.
Forensic Science: Silver Screen vs. Real World
CSI, Law and Order and Bones’ investigators solve murders, robberies and more in merely 60 minutes. Evidence collection proceeds without difficulty. Flashy technology analyzes evidence with ease. Suspects inevitably incriminate themselves. Juries seem to understand expert witness testimony. The evidence positively does or does not match a suspect. The court swiftly reaches a verdict. Justice always prevails.
Actual cases are drastically different from fictional portrayals. Determining specific events leading up to a crime and bringing the correct person to justice is not nearly as simple as on CSI.
Real crime scenes are messy and evidence can be hard to come by. In reality, DNA is not always present as crime dramas would have you believe. Many crime dramas highlight sophisticated analysis techniques, but resources may not be available or it may not even be appropriate for certain types of evidence. In the courtroom, expert witnesses can struggle to convey evidence analysis results in ways that make sense to a non-expert audience, leaving jurors confused and unsure how to proceed. Often lawyers have to work extra hard to explain that conclusions about suspect identification are not always definitive.
Impacts in the courtroom: The CSI Effect
Exaggerated portrayals of forensic science on television may lead to what is known as The CSI Effect. First reported by USA Today in 2004, it refers to the effect forensic science television programs potentially have on jurors.
While research has not definitively proven the CSI Effect, trends show that today’s jurors are seeking more unmistakable proof. Many jurors are looking for sophisticated science to play a role in every trial, and find circumstantial evidence and eyewitness testimony less reliable.
Crime dramas like CSI suggest that deciding if evidence matches a suspect or not is always conclusive. In reality, probability forms the basis of evidence analysis. This can be hard for jurors to understand.
Many feel that the CSI Effect may be increasing the burden of proof in the courtroom. However, the forensic science community has a unique opportunity to turn public fascination of the criminal justice system into opportunities for training and education.
Forensic scientists, lawyers and judges are looking to researchers like the CSAFE team and NIST to find the best tools for explaining the weight of evidence to non-expert audiences. Together, we can help the public recognize fact from fiction and bring the right person to justice.
Learn more about how CSAFE research is increasing the scientific foundations of forensic evidence.
Witnesses of a crime often share their version of the events that led up to the offense. But often, evidence left at the scene is what helps investigators understand the story. Through careful collection, testing, analysis and interpretation techniques, investigators attempt to piece together the details of what took place.
A commonly found type of evidence is pattern evidence. Pattern evidence results from a specific pattern left by physical contact from an object and a surface, known as an impression or imprint. Impressions are three-dimensional markings such as bullet markings, and imprints are two-dimensional, like a fingerprint.
1. What are the potential sources of pattern evidence?
2. What type of information might we want to learn from pattern evidence?
3. Why is pattern evidence challenging?
Pattern evidence is unique in that it requires comparison of images. At present, human interpretation forms the basis of pattern evidence analysis. This method leaves examiners open for bias, error and subjectivity that can affect their decisions. Results may vary between examiners, and juries often struggle to understand the methods an examiner used to arrive at a conclusion.
4. Why do we need to study pattern evidence?
Due to the current subjectivity inherent in the analysis of pattern evidence, organizations like CSAFE are working to develop standard, reproducible ways of analyzing evidence, and to propose effective ways of communicating results. Through additional research, new tools can be built to take more precise measurements even under unfavorable circumstances, we can better understand the effects of time and environmental factors on evidence, as well as determine what characteristics make impression evidence unique when compared to all other types of evidence.
Learn more about CSAFE research in pattern evidence.
When investigating and communicating the value of evidence, the forensic science community makes every effort to explain results to juries in objective and understandable ways.
The analysis of evidence involves the consideration of multiple hypotheses. Let us take DNA evidence for example – What is the probability of observing a genotypic match if the biological sample was left by the suspect? How likely is a match if someone else left the sample?
When quantifying the results of DNA analysis, forensic science experts often use the likelihood ratio. This technique enables experts to communicate the weight of the evidence with a single number.
Forensic DNA analysis rests on a firm foundation of biology, and the public often understands principles such as genetic inheritance and unique combinations of genetic markers. Thus, confidence in the power of DNA evidence to identify an individual is strong.
Forensic scientists have long since used the likelihood ratio effectively in cases where DNA evidence is prominent. We know how biology works and can successfully use statistics to quantify the strength of the evidence.
But what about other types of evidence? Can the likelihood help us to quantify the value of evidence such as fingerprints or bullet striations or the chemical composition of glass? That is still up for debate.
National Institute of Standards and Technology (NIST) statisticians Steve Lund and Hari Iyer recently discussed the value of applying the likelihood ratio in the courtroom. A review of their conclusions is available in a new article published in the NIST Journal of Research.
Evidence analysis and the likelihood ratio applied to DNA evidence is often cut and dry. However, we can’t say the same for evidence such as bullets or fingerprints. Why is that? It is because the fundamental scientific underpinnings of DNA evidence are absent almost everywhere else. We do not have a physical or statistical model for how striations on bullets arise, and we don’t know what determines specific patterns in fingerprints.
The question then becomes-is it still appropriate to use the same statistical tool in areas such as pattern evidence when our level of understanding is not the same? According to Lund and Iyer, the answer is maybe not. Or rather, not yet.
Because in those cases, the likelihood ratio may rest on assumptions that cannot be verified in practice, a degree of subjectivity is inevitable and it may happen that two experts arrive at different values of the likelihood ratio. Thus, say Lund and Iyer, when a decision involves human judgment, caution is key.
Without established models on which to base conclusions in evidence beyond DNA, jurors may find it difficult to interpret the results of forensic analysis. Jurors need more information that just the value of a likelihood ratio, Lund and Iyer say.
Lund and Iyer advocate for complete transparency when explaining the value of evidence to members of a jury. In this light, it is imperative that jurors understand every aspect of the analysis of a specific evidence, and of the methods that were used to draw conclusions. Forensic scientists must also communicate their level of uncertainty in their results. Armed with additional context, jurors are then better prepared to make informed decisions.
As the groundwork for the statistical treatment of evidence is being laid, Lund and Iyer issue a challenge to the forensic science community. When communicating the value of evidence in the courtroom, they call for research that does not focus exclusively on the likelihood ratio approach. More work needs to be done, but they are confident that it is possible to develop more than one framework for the analysis of evidence other than DNA.
Review this NIST news article for more information. Visit the NIST website to discover more forensic science research.