Category: Forensic Science

Understanding How Statistics Can Address Forensic Science Challenges

This post is based on the article “Statistical Issues in Forensic Science” by CSAFE Co-Director Hal Stern of UCI. The full article was published in March 2017.

The way that forensic evidence and other expert testimony is presented in court is determined by legal precedent, especially the 1993 U.S. Supreme Court decision in Daubert v Merrell Dow Pharmaceuticals, and Rule 702 of the Federal Rules of Evidence. These sources identify the trial judge as the “gatekeeper” to identify which testimony to allow, and set the expectation that such testimony “is based on sufficient facts or data” and “is the product of reliable principles and methods”. A 2009 National Research Council (NRC) report and a 2016 report of the President’s Council of Advisors on Science and Technology (PCAST) question whether a number of forensic science disciplines satisfy these expectations. Each calls for additional research to support the science underlying the analysis of forensic evidence.

A multidisciplinary approach is required to carry out such research because each discipline requires technical expertise to collect and prepare evidence and additional expertise to analyze and interpret the resulting data. Statistics has a crucial role to play in helping to address the challenges of forensic science. Examples of the way that statisticians and the field of statistics can contribute are described below.

Reliability and validity

Perhaps the most important challenge facing forensic science after the 2009 NRC and 2016 PCAST reports is the need for data that assesses the reliability and validity of forensic examinations and conclusions. Reliability refers to whether forensic measurements or judgements would be obtained consistently. This includes considerations of repeatability, in which one asks if the same forensic examiner would draw the same conclusion if presented with the same evidence at a different time, and considerations of reproducibility, in which one asks if the same conclusion or measurement would be obtained by other examiners. Reliability is an important first consideration but even if forensic examinations in a particular domain are reliable, that does not indicate whether they are valid or accurate. If a fingerprint examiner concludes that a latent print at the crime scene comes from the same source as a test impression made by the suspect (this conclusion is sometimes known as an identification), we need to know how accurate that conclusion is to make an informed judgment about the weight of evidence. The PCAST report carefully discussed what should be expected in validation studies (e.g., the true status of the samples should be known, the samples should be representative of casework, etc.).

PCAST indicated that statistical methods are well developed and have been validated for single source DNA evidence (or simple mixtures) and for latent print analysis, but reported that additional studies are needed regarding the validity of forensic examinations in other forensic science disciplines. Assessing the reliability and validity of forensic examinations is a key step in developing sound science to measure the strength of evidence and statisticians can contribute to the design and analysis of such studies.

Task-relevant information and cognitive biases

Scientists are humans, not robots. As such, the field of forensic science, as any other domain involving human judgements, has the potential to be compromised by cognitivebias and other cognitive factors which can influence how data is collected, analyzed and findings are communicated. Cognitive bias is a term meant that refers to disconnects between observed behavior, and what would be considered by most as rational decision making.

There are a number of well-known cognitive biases that have been identified in other areas of human decision-making. One example is known as confirmation bias, where examiners may tend to favor interpretations that confirm their own preconceptions about a suspect or the evidence. Another example are framing effects, where a forensic examiner may evaluate evidence differently depending on what contextual information about the suspect or the crime scene was provided.

Statisticians can play a role in teams that study cognitive bias and in teams that try to determine what information should be deemed relevant for a particular task.

Causal inference

Forensic examiners working in crime scene investigation, arson and blood spatter analysis attempt to reconstruct the events leading up to a crime based on evidence found at the crime scene. This process is an effort to infer the causes of observed effects. These examiners face a challenging task because it is difficult to carry out realistic controlled experiments that would allow one to reliably distinguish between competing explanations. For example, the problem of determining if a fire developed naturally or by the use of an accelerant.

Statistical collaboration with practitioners in relevant disciplines will be valuable in strengthening inferences in these settings.

Case processing and procedures

Any process can benefit from careful analysis to understand potential limitations, bottlenecks or sources of error, and the forensic evidence analysis process is no exception.

Crime laboratories are often faced with an overwhelming workload, leading to backlogs that can slow the treatment of evidence. At the same time, maintaining quality forensic examinations requires a quality assurance program that incorporates reanalysis of evidence or verification of conclusions.

Statistical methods can play a role in designing quality assurance programs that can improve efficiency of lab operations while simultaneously insuring the accuracy of conclusions.

Testifying on forensic evidence

Appropriate ways for presenting forensic evidence analysis in the courtroom is an important area of research. Several studies demonstrate the difficulty that jurors can have in understanding statistical ideas like the likelihood ratio and Bayes factor. The forensic science research community is still examining the best way to presnt such evidence in the courtroom.

Statisticians have an important role to play in developing approaches to presenting quantitative evaluations of evidence and in the design and analysis of juror studies to assess the effectiveness of alternative approaches.

How CSAFE Research is Advancing the Field of Forensics

Statisticians at the Center for Statistics and Applications in Forensic Evidence (CSAFE) work in partnership with NIST researchers, forensic science practitioners and scientists in a variety of disciplines to address the issues raised in the NRC and PCAST reports.

Our team is contributing to strengthening the statistical foundations of pattern and digital evidence through the study of existing methods and the development of new statistical to analyze and interpret data and evidence. Our goal is to help forensic scientists in their pursuit of reliable and accurate analyses of forensic evidence.

Learn more about our research.

This post is based on the article “Statistical Issues in Forensic Science” by CSAFE Co-Director Hal Stern of UCI. The full article was published in March 2017.

Forensic Evidence and Forensic Examinations – The Basics

This post is based on the article “Statistical Issues in Forensic Science” by CSAFE Co-Director and Professor of Statistics Hal Stern of Univeristy of California, Irive. Review the full article in the Annual Review of Statistics and Its Application.

When it comes to solving crimes, forensic examiners play a key role in the investigation process and trial proceedings. Their responsibilities vary widely depending on the type of crime and available evidence.

For example, examiners may be called on to identify time of death in a homicide case, reconstruct the events of crime from blood spatter patterns, or take a close look at physical evidence such as broken glass or fingerprints found at the scene. Their analysis of evidence provides crucial details that help in the search for the culprit. The remainder of the discussion here is focused on the role of forensic examiners in attempting to identify if a suspect is linked to particular item of evidence in the case.

It is important to understand that the goal in examining a single piece of evidence is actually not to determine the guilt or innocence of the suspect. Rather, it is an effort to identify if the evidence is associated with the suspect, e.g., if the suspect is a possible source of a fingerprint found at a crime scene. Any number of legitimate reasons could lead to finding evidence from a suspect at the crime scene, such as a visit to the scene at a different time.

Exploring the Different Types of Evidence

DNA – DNA found in biological samples recovered at the crime scene is among today’s most powerful forms of forensic evidence. The forensic examiner’s task when analyzing this type of evidence is to determine if the DNA profile of a suspect matches the DNA profile found in the crime scene sample. They then must assess the significance of this agreement.
Trace Evidence – Refers to evidence types characterized as a fragment or sample of a larger object that is left behind at the time of the crime. This type of evidence could include glass fragments from a broken window, hairs from an individual, or fibers from clothing or carpet. The challenge here is to determine if a sample of trace evidence from the crime scene matches another sample obtained from a suspect (or perhaps from an object in the suspect’s possession).
Pattern evidence – Evidence left at the crime scene that is the result of an impression left by a person or object is known as pattern evidence. The forensic examiner must attempt to determine if the impression found at the crime scene matches the pattern of an analogous sample obtained from the suspect or from an object known to belong to the suspect. Types of pattern evidence include fingerprints, shoeprints, documents/handwriting, toolmarks and firearm impressions.
Digital evidence – Digital technology is playing a larger and larger role in criminal and civil investigations. Digital evidence can refer to any information obtained from a device implicated in an investigation. Examples include images or messages found on a smartphone belonging to a suspect. Digital evidence can be challenging to analyze due to the wide variety of different evidence types that may be found on digital devices.

Recent Events Have Raised Concerns About Current Methods

Forensic examiners summarize their evidence analysis in reports that play a crucial role in investigations and trials. Recent events have called into question the scientific and statistical foundations of evidence analysis and interpretation.

This includes the release of two reports, the 2009 National Research Council (NRC) report and a 2016 report of the President’s Council of Advisors on Science and Technology (PCAST), in which committees comprised of scientists from a number of disciplines questioned whether there was sufficient quantitative data supporting the statements made by forensic examiners in a number of disciplines.

Beyond these reports, an additional source of concern is a number of cases in which forensic science errors have been identified. A famous example is the very public case involving American lawyer Brandon Mayfield. Fingerprint examiners from the Federal Bureau of Investigation (FBI) mistakenly identified Mayfield as the source of a latent fingerprint found at the scene of a 2004 train bombing in Spain.

The Innocence Project, a nonprofit legal organization founded in 1992, details countless other situations where unreliable or improper forensic science led to wrongful convictions. The organization has been instrumental in freeing more than 300 wrongfully convicted individuals through the beginning of 2018 and improper forensic science is identified as a contributor in roughly half of these cases.

Why We Need Statistics In Forensic Evidence Analysis

Both the 2009 NRC report and the 2016 PCAST report emphasized the need for additional study of forensic science methods. It is clear that statistical methods have a key role to play in strengthening the scientific foundations of forensic examinations. Statistics is the science concerned with designing studies and experiments, analyzing and interpreting the results, and summarizing the information obtained. As such, it can contribute to studies aimed at determining the accuracy of the conclusions drawn by forensic examiners, addressing cognitive biases, examining the influence of irrelevant information on analysis, evaluating modifications to case processing procedures and more.

This post is based on the article “Statistical Issues in Forensic Science” by CSAFE Co-Director Hal Stern of UCI published in Annual Review of Statistics and Its Application.

Forensic Terminology Explained: OSAC Releases New Online Lexicon

Organization of Scientific Area Committees (OSAC) for Forensic Science

Forensic science is a broad field that encompasses a wide variety of disciplines and the roles of forensic examiners are diverse. Each discipline and role comes with its own set of vocabulary, which can quickly become confusing. One word may mean one thing to a DNA analyst, but mean something completely different to a handwriting examiner. The Organization of Scientific Area Committees for Forensic Science (OSAC) has developed a new tool to help forensic scientists speak the same language.

If you are searching for what the word identification really means in forensic science, or want to gain a better understanding of cognitive bias, visit the OSAC Lexicon of Forensic Science Terminology.

The OSAC Lexicon Initiative began in 2016 when OSAC’s Forensic Science Standards Board asked each OSAC unit to identify and collect existing terminology related to their specific forensic science discipline. This database of vocabulary contains 4,000 terms organized by forensic discipline. Users can search by discipline and keyword, using either the term or definition and terms are browse-able by letter. Data is exportable to a CSV file.

Often many definitions exist for the same word, and clicking on the record reveals the source for each term. Readers can rest assured that the information comes from a trusted source. The terms and definitions come from the published literature, including documentary standards, specialized dictionaries, Scientific Working Group (SWG) documents, books, journal articles, and technical reports. In addition, the OSAC subcommittees and committees generated or modified many definitions.

OSAC’s goal for 2018 is to add new terms, remove terms, consolidate duplicate entries, verify sources of non-verified terms, and reach consensus on more OSAC Preferred Terms. Users may suggest a term or submit a comment as the Lexicon continues to evolve.

The Lexicon is a big step forward as OSAC seeks to increase clear communication between forensic scientists.

Digital Evidence: How Technology Changes the World of Crime

Our world today revolves around technology more than ever before. From cell phones to tablets to the latest social media application and beyond, countless opportunities exist for digital communication and entertainment.

The Unique Role of Digital Evidence in Forensic Science

The rise of the digital age has introduced a completely new type of crime. Did you know that every digital device such as a mobile phone or cloud storage is a forensic evidence generator? Each device produces mountains of data, carefully recording our every move. Today, emails, financial transactions, photographs, website browser histories and more can reveal key details in a criminal investigation.

Digital evidence involves a unique investigation process. Other types of forensic evidence analysis like shoeprints or fingerprints focus on comparing if the print left at a crime scene matches that of a suspect.

When forensic examiners investigate digital crime, they turn their attention instead to information that exposes the actions and behavior of the individual.

Key Questions Examiners Ask When Analyzing Digital Evidence

Examiners are interested in exploring a suspect’s intent and motive, and search for clues about a suspect’s location and relationship to the victim or others involved in the crime. To find the answers, examiners must ask several questions when analyzing digital data.

For example:

Where is the data stored?
Who was using the device?
Where did the crime take place?
Was the information damaged, destroyed or altered?
Are there deleted files that need to be recovered?

Digital evidence analysis focuses on finding the data stored on the device, and making sense of that information. The challenge is using analysis tools correctly, properly interpreting the results and proving the relationship between the individual and evidential data about a criminal activity.

CSAFE Answers the Call to Find New Solutions to Solve Digital Crime

As the digital world continues to change, there is an ever-increasing need for new techniques that allow efficient, accurate analysis and processing of digital evidence.

CSAFE brings together a collaborative task force of statisticians, machine learning researchers, forensic examiners and more to develop statistically based analytical techniques to improve digital evidence investigations. Learn more about our specific research goals on the CSAFE website.

Our team is investigating a wide range of digital evidence areas, from steganography which analyzes digital photos potentially containing hidden content to using mobile apps to solve digital crime. Review a recent CSAFE Center Wide webinar to see how our statistical approach to time-series of user-generated events captured on digital devices is answering questions about this type of data.

In an effort to provide digital evidence examiners with more tools to conduct digital evidence investigations, CSAFE work in digital evidence continues to expand. Discover how our new projects on digital crime scene reconstruction in cyber bulling and investigation of anonymous marketplaces on the dark web is addressing critical needs in digital crime.

As CSAFE and researchers across the country search for better solutions, we are guided by The Organization of Scientific Area Committees for Forensic Science’s (OSAC) recently released report on digital and multimedia evidence in an effort to increase dialog and harmonize core forensic processes across disciplines.

Partner With Us to Improve Digital Evidence Analysis

Are you a digital evidence examiner or do you have research expertise in this area? We want to hear from you! Learn more about our collaboration opportunities and contact us to join us in improving the scientific foundations of digital evidence analysis.

Human Factors in Forensic Science: CSAFE Researchers Explain Cognitive Psychology’s Role

Is the field of science-based solely on logic and facts? A closer look reveals that science is not always as systematic and foolproof as we may think.

Scientists are humans, not robots. As such, the field of science has the potential to be compromised by human error, bias and other cognitive factors.

CSAFE researchers and psychologists Dr. Daniel Murrie and Dr. Sharon Kelley from the University of Virginia describe how human factors can influence the accuracy of scientific work as a whole, and the specific impact they can have on forensic science.

Defining Human Factors

Dr. Murrie explains that human factors are relevant any time that the human mind or behavior is part of the scientific process. “Human factors, including cognitive bias, can influence how we collect and analyze data, or how we communicate the findings,” he said.

The human brain is powerful and typically steers us in the right direction.

“Our minds are very efficient and that usually serves us well,” Murrie said. “Our brains develop many shortcuts or heuristics that help us make reasonably good decisions fast.”

While this approach to decision making seems practical and beneficial, it has a downside.

“There are a few times when that cognitive infrastructure can lead us to take short cuts or jump to conclusions,” Murrie said. “This can leave us vulnerable to bias in ways that aren’t helpful to science, accuracy or objectivity.”

Human Factors in Forensic Evidence Analysis

The 2009 NAS report calling for reform in forensic science has led to more awareness of human factors and driven new research on cognitive bias in evidence analysis.

For instance, when a fingerprint or shoeprint examiner compares the print of a suspect to a print from the crime scene, there is potential for human factors to bias the examiner’s decision on a match.

Dr. Kelley explains how.

“General social and cognitive psychology research indicates that (A) humans often see what they expect to see and that (B) humans tend to seek out and interpret information in a way that supports their pre-existing beliefs,” she said.

For example, forensic science examiner may have had an opinion about the guilt or innocence of a suspect before even beginning the examination. Whether intentional or not, it could lead to biased results.

Dr. Kelley emphasizes that forensic scientists must be aware of potential bias before expressing opinions or decisions on the outcome of evidence analysis.

Why Is Awareness of Bias, not Enough?

“The good news is that awareness of biases in forensic science is probably at an all-time high. The bad news is that people don’t quite know what to do with this new awareness,” Murrie said.

One concern from recent research is the “bias blind spot,” as cognitive psychologists call it.

“People generally recognize that bias is a problem, but they only see it in others—not themselves,” Murrie said.

Kelley explains that for those who do recognize their own biases, it’s hard to change on your own.

“Introspecting and knowing about the bias and then just trying hard not to be biased is not enough,” Kelley said. “We know from research that these biases often aren’t conscious and you can’t just scan yourself and check for them. Biases still creep into our decision making.”

Instead of squinting your eyes and focusing hard not to be biased, Kelley says procedural and systematic changes are needed.

Strategies to Lessen the Impact of Human Factors in Crime Labs

According to Kelley, crime labs are a very heterogeneous group. They differ in their understanding of cognitive bias and strategies to combat the issue.

So how can researchers like Murrie and Kelley help crime labs reduce the impact of human factors in their investigations?

Two strategies CSAFE is working on to reduce bias are blind verification procedures and context management.

Blind verification procedures serve as checks and balances when analyzing evidence. In this method, a second examiner reviews a case with no information about what the first examiner concluded. Crime laboratories then have two independent decisions to compare. When the two examiners agree, there is more confidence that the analysis is accurate.

Context management involves limiting unnecessary contextual information about the suspect or the crime scene that is irrelevant to the evidence analysis task. For example, when comparing a set of fingerprints, the examiner doesn’t need to know the race or criminal record of the suspect, or even the results of DNA analyses, to do their job. Reducing potentially biasing information increases objectivity in evidence analysis.

Read more about how CSAFE research is addressing context management in crime laboratories.

How Does the Law Impact the Regulation of Forensic Evidence?

Recent advancements in technology have brought forensic science to the forefront in criminal investigations. While investigators are increasingly relying on the scientific foundations of forensic science, constitutional regulation may not be keeping up.

Historically, the U.S. Supreme Court has not taken a firm stance on the type of oversight the constitution should provide concerning forensic science. However, increased reliance on evidence and analysis techniques in the courtroom is persuading the government to take a closer look.

CSAFE researcher and University of Virginia White Burkett Miller Professor of Law and Public Affairs and Justice Thurgood Marshall Distinguished Professor of Law Brandon Garrett recently released a comprehensive review of the latest issues on constitutional regulation of forensic science.

From the article:

“Despite decades of missed opportunities to adequately regulate forensics, in recent rulings the Supreme Court and lower courts increasingly focus on sound litigation of forensics. In an era of plea bargaining, the accuracy of forensic analysis depends far less on cross-examination at trial, and far more on sound lab techniques, full disclosure of strengths and limitations of forensic evidence to prosecutors and the defense, and careful litigation.”

Learn more in the Washington and Lee Law Review journal.

NIST: A Leader in Forensic Science and Valued CSAFE Partner

The entrance sign at NIST's Gaithersburg campus. Credit: J. Stoughton/NIST

Imagine a world without standard measurements. How would we quantify the weight of our food, or the length of material for clothes? Without a standard system, confusion abounds. In 1901 the U.S. government created the National Institute of Standards and Technology (NIST). NIST is a non-regulatory federal agency within the U.S. Department of Justice.

Building the Foundation of Innovation

NIST’s goal is to promote U.S. innovation and industrial competiveness through advances in measurement in science, standards and technology. Innovation relies on the ability to observe and measure which allows product control and standard procedure implementation.

In addition to applications in industry and manufacturing, universally accepted definitions of measurement play a key role in scientific advancement. NIST researchers are at the forefront of developing new technology and commercialization.

“From the smart electronic power grid and electronic health records to atomic clocks, innumerable products and services rely in some way on technology, measurement and standards provided by the National Institute of Standards and Technology.”

Benefits of Standard Measurement in Forensic Science

Did you know that standard measurement is key in forensic science? Universal procedures for collection, analysis and interpretation of forensic evidence help protect the innocent and convict the guilty. NIST focuses on three components of improving forensic science standards.

Science- Leads the way in innovative research in many forensic science disciplines to include DNA, fingerprints, digital evidence and more. Experts increase forensic laboratory analytical method validity through physical reference standards and data.
Policy- Co-chaired the National Commission on Forensic Science to raise awareness on standards such as accreditation requirements for forensic science service providers.
Practice- Directs Organization of Scientific Area Committees for Forensic Science to develop science based standards and guidelines for forensic science disciplines.

CSAFE and NIST- Working together to Transform Forensic Science

NIST is committed to advancing efforts to solve deficiencies in forensic science standards. In 2015, NIST established CSAFE, a Center for Excellence in forensic science. NIST continues to provide funding and strategical guidance to CSAFE as the center works to build the scientific foundations of forensic science. The CSAFE team benefits greatly from the expertise NIST brings to CSAFE research and education initiatives. Working in close partnership, NIST and CSAFE are developing solutions to better recognize, collect, analyze and interpret evidence.

Sources:

https://www.nist.gov/speech-testimony/importance-basic-research-united-states-competitiveness

https://www.nist.gov/about-nist

https://www.nist.gov/topics/forensic-science

A Look Inside the CSI Effect

Are crime scene dramas to blame for unrealistic perceptions of forensic science in the general public?

CBS launched the first episode of CSI: Crime Scene Investigation in the year 2000, and suddenly forensic science was the latest craze. What was once a foreign field to many Americans was now at the forefront of our nation’s curiosity. Nearly 20 years later, viewers are still on the edge of their seats, captivated by stories of crime.

Two important questions to consider:

Does Hollywood get it right?
How are crime dramas influencing real life investigations?

While it may come as a surprise to the everyday citizen, the forensic science community wants you to know that the answers are Hollywood is usually wrong, and television influences public perception more than you might think.

Forensic Science: Silver Screen vs. Real World

CSI, Law and Order and Bones’ investigators solve murders, robberies and more in merely 60 minutes. Evidence collection proceeds without difficulty. Flashy technology analyzes evidence with ease. Suspects inevitably incriminate themselves. Juries seem to understand expert witness testimony. The evidence positively does or does not match a suspect. The court swiftly reaches a verdict. Justice always prevails.

Actual cases are drastically different from fictional portrayals. Determining specific events leading up to a crime and bringing the correct person to justice is not nearly as simple as on CSI.

Real crime scenes are messy and evidence can be hard to come by. In reality, DNA is not always present as crime dramas would have you believe. Many crime dramas highlight sophisticated analysis techniques, but resources may not be available or it may not even be appropriate for certain types of evidence. In the courtroom, expert witnesses can struggle to convey evidence analysis results in ways that make sense to a non-expert audience, leaving jurors confused and unsure how to proceed. Often lawyers have to work extra hard to explain that conclusions about suspect identification are not always definitive.

Impacts in the courtroom: The CSI Effect

Exaggerated portrayals of forensic science on television may lead to what is known as The CSI Effect. First reported by USA Today in 2004, it refers to the effect forensic science television programs potentially have on jurors.

While research has not definitively proven the CSI Effect, trends show that today’s jurors are seeking more unmistakable proof. Many jurors are looking for sophisticated science to play a role in every trial, and find circumstantial evidence and eyewitness testimony less reliable.

Crime dramas like CSI suggest that deciding if evidence matches a suspect or not is always conclusive. In reality, probability forms the basis of evidence analysis. This can be hard for jurors to understand.

Many feel that the CSI Effect may be increasing the burden of proof in the courtroom. However, the forensic science community has a unique opportunity to turn public fascination of the criminal justice system into opportunities for training and education.

Forensic scientists, lawyers and judges are looking to researchers like the CSAFE team and NIST to find the best tools for explaining the weight of evidence to non-expert audiences. Together, we can help the public recognize fact from fiction and bring the right person to justice.

Learn more about how CSAFE research is increasing the scientific foundations of forensic evidence.

Open Source Software: Applications in Forensics and The Courtroom

This is an invited blog post from Richard Torres, an attorney at the Legal Aid Society in New York City. Guest blog posts do not necessarily reflect the views of CSAFE. CSAFE is highlighting this topic due our team’s commitment to the development of open source software and its role in increasing the fair administration of justice.

There is no national oversight body to protect our courts from bad forensic science. The scientific community is the only gatekeeper between questionable forensic practices and the courts, but scientists are not able to examine what they cannot see.

Forensic science software responsible for analyzing evidence is often built with closed source code. This prevents the public, scientists and ultimately, the legal system from being able to review the methods (or code) used to derive scientific conclusions – conclusions that are of utmost importance when someone’s freedom, and in some cases life, is on the line.

Software tools with open source, publically available code allow the scientific and legal communities to conduct a fair review of the processes and steps used to evaluate evidence. The case study below demonstrates the need for greater use of and access to open source software within the criminal justice system.

The Necessity of Open Source Code: A Case Study

The New York City Office of the Chief Medical Examiner (OCME) was one of the first crime laboratories to implement probabilistic genotyping. Probabilistic genotyping is a recent approach for interpreting complex DNA mixtures, many otherwise uninterpretable, and assigning them a statistical weight. It seems that most forensic labs are heading in this direction.

OCME was early in the probabilistic genotyping game when they created a program called the Forensic Statistical Tool (FST). The OCME, with the backing of the five New York City District Attorneys, fought hard to keep the source code for FST behind a tightly closed door even though it was created using tax payers’ dollars.

FST was brought online around 2011. For over six years, OCME lab analysts would testify under oath as to FST results but were unable to explain how FST got its answer. Most judges and jurors did not appear particularly concerned – people trust computers.

My office, The Legal Aid Society, repeatedly sought access to the FST source code in cases where our clients were facing years, even decades in prison. We need to know if faulty forensic science is being used to imprison our clients. Unfortunately, in state court, judges simply would not force OCME to share their code.

A lawyer in our office, Clinton Hughes, decided to build an open source program using the same exact calculations as FST called reQBT. He recruited college interns with math and computer science backgrounds to assist with the finer details – we are lawyers, not scientists. He ran reQBT on FST’s validation study data and, in many cases, reQBT got the same answers. Clint’s team kept reviewing reQBT’s code and could not figure out why reQBT was getting different answers in some cases. We had growing concerns over whether the problem was with reQBT. Was FST’s computer code implementing OCME’s biological models reliably?

In late 2016, as FST was being replaced with a newer program called STRmix, Chris Flood and Sylvie Levine from the New York Federal Defenders’ Office were able to convince Federal Judge Valerie Caproni to order OCME to provide the defense with the FST source code. The Federal Defenders hired Nathan Adams of Forensic Bioinformatics to perform a code review – this was the first time a defense expert reviewed the FST source code. Our suspicion that FST was not properly implementing OCME’s models was confirmed. Adams found that FST performed its calculations differently than what OCME said and it affected FST’s results. There was a catch. Judge Caproni signed a court order preventing Adams from disclosing the specifics of the code problems. FST remained mostly in the dark.

OCME did not immediately respond to Adams alleging in federal court that FST was performing different calculations from what OCME claimed. There were no letters to defense attorneys stating that there may be a problem with FST. It appears that the journals that published the FST studies were not notified either. Were state prosecutors notified?

We renewed our requests to state court judges to order the source code given what Adams found. Ultimately, OCME employees conceded that there was a code change that affected FST’s calculations – possibly a different code change from the one found by Adams. Yet, there was no full scale validation study to establish how well the calculation changes worked.

The Legal Aid Society and Federal Defenders lodged a complaint to the New York State Inspector General requesting an investigation into the changes to FST’s code alongside other concerns about OCME. FST was used in over a thousand cases. Will they need to be reopened? Are people in prison based on faulty forensics?

National press recently reported on our complaint to the Inspector General. At that point, OCME did an about face as to open source. They now say they will share the FST source code.

Applying Lessons Learned to All Forensic Science Disciplines

This problem is not unique to DNA software. Access to open source software is critical to the analysis of pattern and digital evidence. Eliminating bias and error in forensic science is only possible through the total transparency open source code allows. Forensic scientists and researchers responsible for developing these tools must be committed to making their methods publicly available. Lawyers and judges need this to be able to ensure the reliability of the adversarial trial process to better ensure that innocent people are not sitting in jail.

More information about the mission of CSAFE can be found on our homepage. Learn more the impact the Legal Aid Society is making on the criminal justice system on their website.

Stronger Together: How Collaborative Research Is Paving the Way for Groundbreaking Innovation in Forensics

Scientific collaborative research is far from a new idea. Research papers rarely have one author. National funding organizations acknowledge the benefits of scientific collaboration. Today’s technologic advancements make working across institutions, across departments, between disciplines easier than ever.

Yet, as the Center for Statistics and Applications in Forensic Evidence, many are led to believe — and easily so — that we are a team composed solely of statisticians. While we do work with numerous statisticians, our research team comprises over 60 distinguished scientists and practitioners across departments and disciplines.

We collaborate with lawyers to analyze and research forensic testimony and evaluation of forensic evidence. We work with computer engineers, some of whom are undertaking mobile app forensic analysis. We have psychologists and behavior analysts on our team who investigate human factors at crime laboratories. We have mechanical, electrical and computer engineers some of whom are undertaking research in fluid dynamics in bloodstain pattern recognition.

How Collaborative Research Across Scientific Fields Expands Our Capabilities

The goals of CSAFE as a whole are to develop statistical foundations to help forensic scientists analyze and interpret evidence with consistent objectivity, and reduce human factors. We strive to help educate the community of forensic practitioners and other key stakeholders on how to communicate these complex results and implications in clear, honest ways. We work to translate groundbreaking research into practical applications for real-world forensic science investigations.

CSAFE’s key research areas require much time and effort, multi-disciplinary knowledge and a large network of people with varying capabilities and capacities.

Collaborative research also allows us to:

Share resources
Expand our scope of research
Lend more credibility to projects
Tackle the complex issues facing the forensic community

By incorporating a diverse group of scientists across fields of study, CSAFE has been able to undertake 29 current projects with others on queue for the future.

“The types of problems CSAFE aims to solve and the research in which we are engaged absolutely requires that we collaborate with scientists and practitioners,” said CSAFE Director Dr. Alicia Carriquiry.

Welcoming Collaborative Research Across All Fields of Science

CSAFE’s collaborative philosophy is simple and inclusive: By bringing together some of the most accomplished scientists and practitioners across scientific disciplines and from around the world, we are better equipped to build strong scientific foundations to apply to real-world forensic science investigations.

We welcome scientists from any field to contact us to discuss how you and your team could help undertake projects in our research areas.