Statistical Methods for Analyzing Event Time-Series Data in Digital Forensics

This CSAFE webinar was presented by Dr. Padhraic Smyth from University of California, Irvine on September 28, 2017.

Time-series of user-generated events are routinely captured and logged on devices such as computers and mobile phones. This type of data is of increasing interest in forensic investigations. There has been relatively little use of statistical thinking in this context, for a variety of reasons. In this talk we will discuss recent work at UC Irvine (under CSAFE) that has begun to develop statistical techniques to answer basic questions about such data, building on methods such as marked point processes. We will outline the methodological approach and describe results to date on both simulated and real-world data. The talk will conclude with a brief discussion of challenges in this area as well as some thoughts on future research directions.

Explaining Source Conclusions to a Lay Audience

This CSAFE Center Wide Meeting Webinar was presented by Dr. William (Bill) Thompson from University of California, Irvine on November 30, 2017.

Forensic scientists are often asked to evaluate whether two items have (or might have) a common source. What is the best way for them to explain the strength of their conclusions in reports and testimony? This presentation will review a number of options, discussing both the logical foundations of each approach and what social science research tells us about how well various presentation formats are likely to be understood by a lay audience. The presentation will focus particularly on the risk that statistical data will be misunderstood or used inappropriately to draw fallacious conclusions, and on possible steps to mitigate that risk.

Open Source Software: Applications in Forensics and The Courtroom

This is an invited blog post from Richard Torres, an attorney at the Legal Aid Society in New York City. Guest blog posts do not necessarily reflect the views of CSAFE. CSAFE is highlighting this topic due our team’s commitment to the development of open source software and its role in increasing the fair administration of justice. 

There is no national oversight body to protect our courts from bad forensic science. The scientific community is the only gatekeeper between questionable forensic practices and the courts, but scientists are not able to examine what they cannot see.

Forensic science software responsible for analyzing evidence is often built with closed source code. This prevents the public, scientists and ultimately, the legal system from being able to review the methods (or code) used to derive scientific conclusions – conclusions that are of utmost importance when someone’s freedom, and in some cases life, is on the line.

Software tools with open source, publically available code allow the scientific and legal communities to conduct a fair review of the processes and steps used to evaluate evidence. The case study below demonstrates the need for greater use of and access to open source software within the criminal justice system.

The Necessity of Open Source Code: A Case Study 

The New York City Office of the Chief Medical Examiner (OCME) was one of the first crime laboratories to implement probabilistic genotyping.  Probabilistic genotyping is a recent approach for interpreting complex DNA mixtures, many otherwise uninterpretable, and assigning them a statistical weight.  It seems that most forensic labs are heading in this direction.

OCME was early in the probabilistic genotyping game when they created a program called the Forensic Statistical Tool (FST).  The OCME, with the backing of the five New York City District Attorneys, fought hard to keep the source code for FST behind a tightly closed door even though it was created using tax payers’ dollars.

FST was brought online around 2011.  For over six years, OCME lab analysts would testify under oath as to FST results but were unable to explain how FST got its answer.  Most judges and jurors did not appear particularly concerned – people trust computers.

My office, The Legal Aid Society, repeatedly sought access to the FST source code in cases where our clients were facing years, even decades in prison.  We need to know if faulty forensic science is being used to imprison our clients.  Unfortunately, in state court, judges simply would not force OCME to share their code.

A lawyer in our office, Clinton Hughes, decided to build an open source program using the same exact calculations as FST called reQBT.  He recruited college interns with math and computer science backgrounds to assist with the finer details – we are lawyers, not scientists.  He ran reQBT on FST’s validation study data and, in many cases, reQBT got the same answers.  Clint’s team kept reviewing reQBT’s code and could not figure out why reQBT was getting different answers in some cases.  We had growing concerns over whether the problem was with reQBT.  Was FST’s computer code implementing OCME’s biological models reliably?

In late 2016, as FST was being replaced with a newer program called STRmix, Chris Flood and Sylvie Levine from the New York Federal Defenders’ Office were able to convince  Federal Judge Valerie Caproni to order OCME to provide the defense with the FST source code.  The Federal Defenders hired Nathan Adams of Forensic Bioinformatics to perform a code review – this was the first time a defense expert reviewed the FST source code.  Our suspicion that FST was not properly implementing OCME’s models was confirmed.  Adams found that FST performed its calculations differently than what OCME said and it affected FST’s results.  There was a catch.  Judge Caproni signed a court order preventing Adams from disclosing the specifics of the code problems.  FST remained mostly in the dark.

OCME did not immediately respond to Adams alleging in federal court that FST was performing different calculations from what OCME claimed.  There were no letters to defense attorneys stating that there may be a problem with FST.  It appears that the journals that published the FST studies were not notified either.  Were state prosecutors notified?

We renewed our requests to state court judges to order the source code given what Adams found.  Ultimately, OCME employees conceded that there was a code change that affected FST’s calculations – possibly a different code change from the one found by Adams.  Yet, there was no full scale validation study to establish how well the calculation changes worked.

The Legal Aid Society and Federal Defenders lodged a complaint to the New York State Inspector General requesting an investigation into the changes to FST’s code alongside other concerns about OCME.  FST was used in over a thousand cases.  Will they need to be reopened?  Are people in prison based on faulty forensics?

National press recently reported on our complaint to the Inspector General.  At that point, OCME did an about face as to open source. They now say they will share the FST source code.

Applying Lessons Learned to All Forensic Science Disciplines

This problem is not unique to DNA software.  Access to open source software is critical to the analysis of pattern and digital evidence.  Eliminating bias and error in forensic science is only possible through the total transparency open source code allows. Forensic scientists and researchers responsible for developing these tools must be committed to making their methods publicly available.  Lawyers and judges need this to be able to ensure the reliability of the adversarial trial process to better ensure that innocent people are not sitting in jail.

More information about the mission of CSAFE can be found on our homepage.  Learn more the impact the Legal Aid Society is making on the criminal justice system on their website.

Forensic Pattern Recognition Evidence-An Educational Module

This learning module (and associated instructor’s guide) was developed by CSAFE researcher and criminology, law, and society professor from University of California, Irvine Dr. Simon Cole.   The educational opportunity uses latent print identification as a case study of the broader category of forensic pattern recognition evidence. This in turn reflects the interaction between science, law, and policy. The module has been explicitly designed for non-scientists from a variety of different backgrounds including law and public policy.  Based on an active learning approach, the module is based  on scientific and legal literature, policy documents, and court exhibits and opinions.  Learn more and view the module on the National Academies of Sciences, Engineering, and Medicine website.

Stronger Together: How Collaborative Research Is Paving the Way for Groundbreaking Innovation in Forensics

Scientific collaborative research is far from a new idea. Research papers rarely have one author. National funding organizations acknowledge the benefits of scientific collaboration. Today’s technologic advancements make working across institutions, across departments, between disciplines easier than ever.

Yet, as the Center for Statistics and Applications in Forensic Evidence, many are led to believe — and easily so — that we are a team composed solely of statisticians. While we do work with numerous statisticians, our research team comprises over 60 distinguished scientists and practitioners across departments and disciplines.

We collaborate with lawyers to analyze and research forensic testimony and evaluation of forensic evidence. We work with computer engineers, some of whom are undertaking mobile app forensic analysis. We have psychologists and behavior analysts on our team who investigate human factors at crime laboratories. We have mechanical, electrical and computer engineers some of whom are undertaking research in fluid dynamics in bloodstain pattern recognition.

How Collaborative Research Across Scientific Fields Expands Our Capabilities

The goals of CSAFE as a whole are to develop statistical foundations to help forensic scientists analyze and interpret evidence with consistent objectivity, and reduce human factors. We strive to help educate the community of forensic practitioners and other key stakeholders on how to communicate these complex results and implications in clear, honest ways. We work to translate groundbreaking research into practical applications for real-world forensic science investigations.

CSAFE’s key research areas require much time and effort, multi-disciplinary knowledge and a large network of people with varying capabilities and capacities.

Collaborative research also allows us to:

  • Share resources
  • Expand our scope of research
  • Lend more credibility to projects
  • Tackle the complex issues facing the forensic community

By incorporating a diverse group of scientists across fields of study, CSAFE has been able to undertake 29 current projects with others on queue for the future.

“The types of problems CSAFE aims to solve and the research in which we are engaged absolutely requires that we collaborate with scientists and practitioners,” said CSAFE Director Dr. Alicia Carriquiry.

Welcoming Collaborative Research Across All Fields of Science

CSAFE’s collaborative philosophy is simple and inclusive: By bringing together some of the most accomplished scientists and practitioners across scientific disciplines and from around the world, we are better equipped to build strong scientific foundations to apply to real-world forensic science investigations.

We welcome scientists from any field to contact us to discuss how you and your team could help undertake projects in our research areas.

Forensic Statistics for Lawyers Part III: Comparison/Identification

CSAFE researcher and faculty member at the University of California, Irvine Bill Thompson has developed a series of videos as a Lawyer’s Guide to Probability and Statistics in Forensic Science.  The goal of these videos is to help lawyers understand the numbers that forensic scientists use to characterize the strength of evidence.  It’s also designed to help lawyers understand non-numerical statements that forensic scientists might make that are based on a statistical or probabilistic analysis.

Part III: Comparison/Identification

Forensic Statistics for Lawyers Part II: Classification

CSAFE researcher and faculty member at the University of California, Irvine Bill Thompson has developed a series of videos as a Lawyer’s Guide to Probability and Statistics in Forensic Science.  The goal of these videos is to help lawyers understand the numbers that forensic scientists use to characterize the strength of evidence.  It’s also designed to help lawyers understand non-numerical statements that forensic scientists might make that are based on a statistical or probabilistic analysis.

Part II: Classification

Forensic Statistics for Lawyers Part I: Quantification

CSAFE researcher and faculty member at the University of California, Irvine Bill Thompson has developed a series of videos as a Lawyer’s Guide to Probability and Statistics in Forensic Science.  The goal of these videos is to help lawyers understand the numbers that forensic scientists use to characterize the strength of evidence.  It’s also designed to help lawyers understand non-numerical statements that forensic scientists might make that are based on a statistical or probabilistic analysis.

Part I: Quantification

Get to Know the Different Types of Forensic Scientists

At the Center for Statistics and Applications in Forensic Evidence (CSAFE), we often discuss the community of forensic scientists who drive our efforts to increase the scientific foundation of forensic science. Some may wonder, though, who are these forensic scientists? What do they do?

Unlike modern-day stereotypes of the forensic science field, forensic scientists cannot solve crimes in an hour’s time with absolute certainty. Forensic scientists of all types must work together to collect, analyze and interpret digital and pattern evidence, with the goal to bring the correct person to justice in a criminal investigation.

While “collecting,” “analyzing” and “interpreting” are easy words to say, they are far from simple in practice. Forensic scientists must sift through crime scenes for evidence deemed relevant to the crime. They analyze every inch of that evidence, look at it from all angles, and apply scientific principals and tests to understand that evidence’s significance and culpability. Even then, the results do not manifest in a black and white answer. Rarely does evidence provide 100 percent certainty — and nor can forensic scientists claim that their work is 100 percent error- and bias-free. That is where CSAFE’s statistical models support practitioners in the challenge to increase accuracy.

In the process of analyzing evidence, forensic scientists from different focus areas use their unique set of skills to help bring the correct person to justice. The information below provides an inside look at several common forensic scientist professions. Although the duties of each specialty differ and may seem unrelated, each plays a vital role in the discovery of the facts related to a crime.

Criminalist or Forensic Scientist: This forensic scientist works to reconstruct crime scenes in order to collect evidence. They analyze the results with the goal of determining the circumstances that led up to the crime and potential motives of the suspect. Their work requires a broad range of knowledge in scientific principles and effective deductive reasoning skills.

Blood Splatter Analyst: Experts assess the diameter and shape of bloodstains, which reflect the origin and trajectory of external blood flow. This professional works to help provide important clues as to the victim’s movements at the time of an assault, and movement of the person’s body afterwards.

Firearms Examiner: Professionals in this field analyze guns and weapons that may have been used to commit crimes. This professional may fire test shots from a certain weapon in order to compare bullet fragments or shell casings. They also may present ballistic evidence and their findings in court as an expert witness.

Digital Forensic Investigator: This is a type of forensic scientist who focuses on cyber crimes and works to discover evidence from computers, mobile devices, cloud computing, and electronic storage. This professional works within IT systems to extract data that can provide insights into the elements of a crime.

Forensic Anthropologist: Specialists study human bones and skeletal structures to determine age, gender, race and other identifying characteristics of a victim of a crime.

Pathologist: As a trained medical doctor, this person performs autopsies to assist with postmortem identification and to determine the cause of death of a victim of a crime. This professional may visit the scene of a crime to help investigators determine circumstances leading up to the fatality.

Questioned Document Examiner: This is a forensic scientist who analyzes handwriting and signatures for authenticity. They may also restore, decipher and assess missing pieces and pages of documents when needed.

Toxicologist: Combining the disciplines of toxicology, analytical chemistry, pharmacology and clinical chemistry, this scientist analyzes bodily fluids, hair, and nails for chemical substances to aid in investigations related to poisoning and drug use.

Forensic Psychologist: This specialty focuses on profiling persons involved in a crime through interviewing suspects, victims, and witnesses to help determine motives and mental states. They focus on issues such as competence to stand trial, testify or make medical decisions. They also assess mental illness as it relates to criminal responsibility,

Professor of Forensic Science: Working to to educate the next generation of forensic scientists, this professional typically has real world experience in the field, and helps students build knowledge of forensic science guided by scientific principles. Professors may teach undergraduate students, or work with graduate students pursuing a Master’s or Ph.D. degree. They may also lead a research group and study different areas of forensics.

Collectively, these scientists pool their efforts to use statistical foundations to reduce speculations in the analyzation of evidence and to limit human factors that influence decision making. Forensic practitioners work toward a common goal of communicating findings in clear, effective ways to the broader forensic and judicial communities. The mission of CSAFE is to align itself with these same goals and to build on the skills each team member brings to advance the field of forensics. To collaborate with CSAFE and help us accomplish these goals, contact us here.








Statistical and Algorithmic Approaches to Matching Bullets

This CSAFE Center Wide Meeting Webinar was presented by Eric Hare from Iowa State University on April 14, 2017.
CSAFE researchers at Iowa State University will discuss the advances they have made towards providing firearms examiners with an objective, quantifiable and standard approach to describe the degree of similarity observed between two bullets. Researchers will explain how 3D images of bullet signatures are being used to develop computer assisted algorithms to look at which features hold the most information and enable discrimination between samples. Progress towards developing a signature-based score that quantifies the differences between bullets will also be discussed.