A recently published analysis outlines the challenges of implementing blind proficiency tests in forensic labs and the potential solutions to overcome them.
Proficiency testing is required at accredited forensic labs. However, a 2014 survey found that 98 percent of forensic labs reported conducting some kind of proficiency test, but only 10 percent conducted blind proficiency tests. That rate had held steady for a decade.
In November 2018, the Center for Statistics and Applications in Forensic Evidence (CSAFE) hosted a meeting to discuss how blind testing may be implemented more frequently. Attendees discussed the challenges labs may face in adopting the practice, and the possible solutions for these challenges.
The resulting paper, published in Forensic Science International: Synergy, drew from those discussions CSAFE researchers had with lab directors and quality managers from seven forensic lab systems in the eastern U.S. and the Houston Forensic Science Center.
Robin Mejia, CSAFE co-director and one of the authors of the study, said blind proficiency testing is an important tool to add to forensic quality assurance programs because it can demonstrate quality and provide the opportunity to identify issues in the entire laboratory case processing pipeline.
“With declared tests, examiners know they are being tested, whereas with blind testing, they believe they are analyzing real casework and don’t know they’re being tested,” she said. “Blind tests have several benefits. For the program to work, the tests must look like the actual casework examiners see every day. Plus, people can behave differently when they know they’re taking a test. Blind testing allows the lab to demonstrate the quality of work they do on regular cases, where analysts don’t know they’re being tested.”
Mejia noted that forensic labs face many challenges in implementing a blind proficiency testing program, and participants at the meeting identified the problems they had encountered and if and how they solved them. Many individuals came from labs that had not implemented blind testing at all.
The researchers extracted eight core challenges from the discussions, including creating realistic test cases and submission materials, developing procedures to ensure test results are not released as real cases, the financial costs of implementing the tests and determining the metrics to collect and report. The paper offered solutions or the steps needed to move forward with implementation and identified key stakeholders who can help make blind testing possible for more laboratories.
“The potential benefits of implementing blind proficiency testing would justify the investment of resources to make it a standard component of forensic quality assurance programs,” said Maria Cuellar, an assistant professor at the University of Pennsylvania and CSAFE researcher who also contributed to the study.
Jeff Salyards, a member of the CSAFE strategic board and a principal analyst at Compass Scientific Consulting, also contributed to the paper.
The CSAFE research team is looking for forensic labs that want to implement blind testing or labs that have implemented blind tests and would be interested in helping other labs advance. Contact Robin Mejia at firstname.lastname@example.org for more information.
View the full journal article here.
View CSAFE’s webinar on implementing blind proficiency testing here.
Check out an additional article on implementing a blind quality control program in blood alcohol analysis here.