This CSAFE webinar was held on May 10, 2022.<\/p>\n
Presenters: <\/strong> Hari Iyer Presentation Description:<\/strong> A given body of validation testing may provide a lot of information about some use cases and little-to-no information about others. Unfortunately, there are often far too many potential use cases to conduct many validation tests for every scenario that occurs in casework. That is, we will generally have little to no validation data for the exact scenario of a given case. This unsettling truth may cause some to wonder whether one can reasonably say anything about the expected performance of an algorithm applied in a given case.<\/p>\n Rather than addressing these concerns with sound science, common phrases like \u201cvalidated\u201d or \u201cestablished error rate\u201d promote an imaginary finish line and sidestep critical considerations of case-specific reliability. What is needed is a paradigm shift; instead of asking whether a method or algorithm has been \u201cvalidated,\u201d we should be asking, \u201cwhat does the available body of validation testing suggest about the performance of the method in the case at hand?\u201d Because performance varies with the type of case considered, this question should be asked for each and every case in which the method is applied. We need an organized, easy-to-follow overview of the information provided by available validation data regarding algorithm performance in the current case.<\/p>\n By modeling method performance using factors that describe a case\u2019s type (e.g., estimated amount of a given contributor\u2019s DNA in a mixture or the number of minutiae detected in a latent fingerprint) and are suspected of affecting difficulty, we order validation tests in terms of difficulty. For a given case, we then find its place in the ordering and assess the performance among contiguous subsets of validation runs less difficult and more difficult than the current case, respectively. These performance characterizations are used to estimate an interval intended to span the method\u2019s performance is situations like the case at hand. Using this approach provides the following critical and easy- to-understand information:<\/p>\n Continually placing the information gathered in validation front and center in reports and testimony focuses attention on the body of empirical results rather than opinions and speculation. Additionally, this framework can help forensic service providers determine and justify whether a given method should be applied to a given case as well as decide whether and where to further invest resources for validation, either in support of a particular case or generally. Unlike a binary notion of \u201cvalidated\u201d methods, this approach reflects the natural benefit of continued testing. When many validation samples are available, one can increasingly restrict the set of uniformly more challenging use cases to those only slightly more challenging than the current case, thereby providing a better \u201cworst-case\u201d performance estimate. This is the appropriate scientific lens through which to view validation testing \u2013 more testing produces less biased results and lower uncertainties. We illustrate the approach described above for the discipline of DNA mixture interpretation using the publicly available ProvedIT database.<\/p>\n Webinar Recording:<\/strong>
\nSteve Lund
\nStatistical Engineering Division, NIST<\/p>\n
\nStatistical Engineering Division, NIST<\/p>\n
\nThe legal and scientific communities agree that any forensic method or algorithm should undergo validation testing before it is relied upon in the judicial system to make life-altering decisions. Validation results are often summarized using average error rates. Given that method performance can vary due to factors like the quantity, quality, and complexity of the questioned impression or sample, case-specific performance assessments are far more relevant than overall average performance assessments.<\/p>\n\n
\n