Skip to content

The Proficiency of Experts

Journal: University of Pennsylvania Law Review
Published: 2017
Primary Author: Brandon Garrett
Secondary Authors: G. Mitchell

Expert evidence plays a crucial role in civil and criminal litigation. Changes in the rules concerning expert admissibility, following the Supreme Court’s Daubert ruling, strengthened judicial review of the reliability and the validity of an expert’s methods. Judges and scholars, however, have neglected the threshold question for expert evidence: whether a person should be qualified as an expert in the first place. Judges traditionally focus on credentials or experience when qualifying experts without regard to whether those criteria are good proxies for true expertise. We argue that credentials and experience are often poor proxies for proficiency. Qualification of an expert presumes that the witness can perform in a particular domain with a proficiency that non-experts cannot achieve, yet many experts cannot provide empirical evidence that they do in fact perform at high levels of proficiency. To demonstrate the importance of proficiency data, we collect and analyze two decades of proficiency testing of latent fingerprint examiners. In this important domain, we found surprisingly high rates of false positive identifications for the period 1995 to 2016. These data would qualify the claims of many fingerprint examiners regarding their near infallibility, but unfortunately, judges do not seek out such information. We survey the federal and state case law and show how judges typically accept expert credentials as a proxy for proficiency in lieu of direct proof of proficiency. Indeed, judges often reject parties’ attempts to obtain and introduce at trial empirical data on an expert’s actual proficiency. We argue that any expert who purports to give falsifiable opinions can be subjected to proficiency testing and that proficiency testing is the only objective means of assessing the accuracy and reliability of experts who rely on subjective judgments to formulate their opinions (so-called “black-box experts”). Judges should use proficiency data to make expert qualification decisions when the data is available, should demand proof of proficiency before qualifying black-box experts, and should admit at trial proficiency data for any qualified expert. We seek to revitalize the standard for qualifying experts: expertise should equal proficiency.

Related Resources

Blind Testing in Firearms Examination: Preliminary Results, Benefits, Limitations, and Future Directions

Blind Testing in Firearms Examination: Preliminary Results, Benefits, Limitations, and Future Directions

This CSAFE webinar was held on June 23, 2022. Presenters: Maddisen Neuman Quality / Research Associate, Houston Forensic Science Center   Presentation Description: Open proficiency tests created by external vendors…
Surveying Practicing Firearm Examiners

Surveying Practicing Firearm Examiners

A sample (n = 79) of practicing firearm and toolmark examiners was queried about casework as well as their views about the potential role that statistics might play in future…
Blind testing in firearms: Preliminary results from a blind quality control program

Blind testing in firearms: Preliminary results from a blind quality control program

Open proficiency tests meet accreditation requirements and measure examiner competence but may not represent actual casework. In December 2015, the Houston Forensic Science Center began a blind quality control program…
Extracting Case-Specific Data from Validation Studies

Extracting Case-Specific Data from Validation Studies

This CSAFE webinar was held on May 10, 2022. Presenters: Steve Lund Statistical Engineering Division, NIST Hari Iyer Statistical Engineering Division, NIST Presentation Description: The legal and scientific communities agree…