This is an invited blog post from Richard Torres, an attorney at the Legal Aid Society in New York City. Guest blog posts do not necessarily reflect the views of CSAFE. CSAFE is highlighting this topic due our team’s commitment to the development of open source software and its role in increasing the fair administration of justice.
There is no national oversight body to protect our courts from bad forensic science. The scientific community is the only gatekeeper between questionable forensic practices and the courts, but scientists are not able to examine what they cannot see.
Forensic science software responsible for analyzing evidence is often built with closed source code. This prevents the public, scientists and ultimately, the legal system from being able to review the methods (or code) used to derive scientific conclusions – conclusions that are of utmost importance when someone’s freedom, and in some cases life, is on the line.
Software tools with open source, publically available code allow the scientific and legal communities to conduct a fair review of the processes and steps used to evaluate evidence. The case study below demonstrates the need for greater use of and access to open source software within the criminal justice system.
The Necessity of Open Source Code: A Case Study
The New York City Office of the Chief Medical Examiner (OCME) was one of the first crime laboratories to implement probabilistic genotyping. Probabilistic genotyping is a recent approach for interpreting complex DNA mixtures, many otherwise uninterpretable, and assigning them a statistical weight. It seems that most forensic labs are heading in this direction.
OCME was early in the probabilistic genotyping game when they created a program called the Forensic Statistical Tool (FST). The OCME, with the backing of the five New York City District Attorneys, fought hard to keep the source code for FST behind a tightly closed door even though it was created using tax payers’ dollars.
FST was brought online around 2011. For over six years, OCME lab analysts would testify under oath as to FST results but were unable to explain how FST got its answer. Most judges and jurors did not appear particularly concerned – people trust computers.
My office, The Legal Aid Society, repeatedly sought access to the FST source code in cases where our clients were facing years, even decades in prison. We need to know if faulty forensic science is being used to imprison our clients. Unfortunately, in state court, judges simply would not force OCME to share their code.
A lawyer in our office, Clinton Hughes, decided to build an open source program using the same exact calculations as FST called reQBT. He recruited college interns with math and computer science backgrounds to assist with the finer details – we are lawyers, not scientists. He ran reQBT on FST’s validation study data and, in many cases, reQBT got the same answers. Clint’s team kept reviewing reQBT’s code and could not figure out why reQBT was getting different answers in some cases. We had growing concerns over whether the problem was with reQBT. Was FST’s computer code implementing OCME’s biological models reliably?
In late 2016, as FST was being replaced with a newer program called STRmix, Chris Flood and Sylvie Levine from the New York Federal Defenders’ Office were able to convince Federal Judge Valerie Caproni to order OCME to provide the defense with the FST source code. The Federal Defenders hired Nathan Adams of Forensic Bioinformatics to perform a code review – this was the first time a defense expert reviewed the FST source code. Our suspicion that FST was not properly implementing OCME’s models was confirmed. Adams found that FST performed its calculations differently than what OCME said and it affected FST’s results. There was a catch. Judge Caproni signed a court order preventing Adams from disclosing the specifics of the code problems. FST remained mostly in the dark.
OCME did not immediately respond to Adams alleging in federal court that FST was performing different calculations from what OCME claimed. There were no letters to defense attorneys stating that there may be a problem with FST. It appears that the journals that published the FST studies were not notified either. Were state prosecutors notified?
We renewed our requests to state court judges to order the source code given what Adams found. Ultimately, OCME employees conceded that there was a code change that affected FST’s calculations – possibly a different code change from the one found by Adams. Yet, there was no full scale validation study to establish how well the calculation changes worked.
The Legal Aid Society and Federal Defenders lodged a complaint to the New York State Inspector General requesting an investigation into the changes to FST’s code alongside other concerns about OCME. FST was used in over a thousand cases. Will they need to be reopened? Are people in prison based on faulty forensics?
National press recently reported on our complaint to the Inspector General. At that point, OCME did an about face as to open source. They now say they will share the FST source code.
Applying Lessons Learned to All Forensic Science Disciplines
This problem is not unique to DNA software. Access to open source software is critical to the analysis of pattern and digital evidence. Eliminating bias and error in forensic science is only possible through the total transparency open source code allows. Forensic scientists and researchers responsible for developing these tools must be committed to making their methods publicly available. Lawyers and judges need this to be able to ensure the reliability of the adversarial trial process to better ensure that innocent people are not sitting in jail.