{"id":8333,"date":"2021-01-20T10:45:48","date_gmt":"2021-01-20T16:45:48","guid":{"rendered":"https:\/\/forensicstats.org\/?post_type=tribe_events&p=8333"},"modified":"2021-04-16T08:29:11","modified_gmt":"2021-04-16T13:29:11","slug":"webinar-irt-for-forensics-a-re-analysis-of-the-fbi-black-box-study","status":"publish","type":"tribe_events","link":"https:\/\/forensicstats.org\/event\/webinar-irt-for-forensics-a-re-analysis-of-the-fbi-black-box-study\/","title":{"rendered":"Webinar: IRT for Forensics: A re-analysis of the FBI \u201cBlack Box\u201d Study"},"content":{"rendered":"

This event took place on April 8, 2021. A recording of the event can be found below.<\/em><\/p>\n

CSAFE invites researchers, collaborators, and members of the broader forensics and statistics communities to participate in our Spring 2021 Webinar Series on Thursday, April 8, 2021, from 1:00pm-2:00pm CST.<\/strong> The presentation will be \u201c<\/strong><\/em>IRT for Forensics: A re-analysis of the FBI \u201cBlack Box\u201d Study.\u201d\u00a0<\/strong><\/em><\/p>\n

Presenter: <\/strong><\/p>\n

Amanda Luby
\nAssistant Professor in Statistics, Swarthmore College<\/p>\n

Presentation Description:<\/strong><\/p>\n

In this webinar, Amanda Luby will explore how Item Response Theory (IRT), a class of statistical methods used prominently in educational testing, can be used to measure participant proficiency in error rate studies. Using the FBI \u201cBlack Box\u201d data, Luby will illustrate the strengths of an IRT-based analysis over traditional \u201cpercent correct\u201d scoring.<\/p>\n

The FBI \u201cBlack Box\u201d Study (Ulery, 2011) was designed to estimate casework error rates of latent print comparisons in the United States, and additional \u201cBlack Box\u201d studies have been called for in other pattern evidence disciplines to estimate error rates. While such studies provide error rate estimates aggregated over all examiners, we cannot directly compare individual examiners\u2019 error rates, since each participant typically is asked to evaluate a random subset of comparison tasks (items) and some items are more difficult than others.<\/p>\n

IRT estimates proficiency among participants while simultaneously accounting for varying difficulty among items. Using an IRT-based analysis, we find that the largest variability in examiner decisions occurs in print quality assessments and inconclusive decisions. We also find some participants were likely to over- or under-report difficulty even after accounting for their proficiency, item difficulty, and other participants\u2019 reported difficulty; and examiners who report items to be more difficult perform similarly to examiners who report items to be easier. These results underscore the importance of better understanding the cognitive factors involved in latent print examination decisions.
\n