Month: February 2022

NIST Releases Results from a Black Box Study for Digital Forensic Examiners

NIST Black Box Study for Digital Forensic Examiners

The National Institute of Standards and Technology (NIST) has published the results from a black box study for digital forensic examiners. The study, released in February 2022, describes the methodology used in the study and summarizes the results.

The study was conducted online and open to anyone in the public or private sectors working in the digital forensics field. Survey participants examined and reported on the simulated digital evidence from casework-like scenarios. NIST said study’s goal was to assess the performance of the digital forensic community as a whole.

Results from a Black-Box Study for Digital Forensic Examiners (NISTIR 8412) can be viewed at https://nvlpubs.nist.gov/nistpubs/ir/2022/NIST.IR.8412.pdf.

From Results from a Black-Box Study for Digital Forensic Examiners, page 33:
Summary Key Takeaways
Despite the limitations of the study, two key takeaways about the state of the digital evidence discipline emerged:
Digital forensics examiners showed that they can answer difficult questions related to the analysis of mobile phones and personal computers. Questions ranged from basic, such as identifying who the user of the phone had contacted, to advanced questions that related to the use of the TOR browser.
The response to the study underscored the size, variety, and complexity of the field. The study received responses from examiners working in international, federal, state, local government, and private labs whose major work included law enforcement, defense, intelligence, and incident response/computer security. There were also responses from people outside of these areas.

Results Available from OSAC Registry Implementation Survey

OSAC Registry Implementation Survey: 2021 Report

The Organization of Scientific Area Committees for Forensic Science (OSAC) released the results from its first annual Registry Implementation Survey. The report, published in February 2022, provides a detailed look at the respondents and the implementation status of the 46 standards represented in the survey.

In the summer of 2021, OSAC released the survey targeted at forensic science service providers from across the country. It was designed to help OSAC better understand how the standards on the OSAC registry are being used, the challenges around standards implementation and what support is needed to improve it.

The OSAC Registry Implementation Survey: 2021 Report is available at https://www.nist.gov/osac/osac-registry-implementation-survey.

From page 10 of the OSAC Registry Implementation Survey: 2021 Report:
Priority for Implementing Standards
When asked what priority survey participants considered standards implementation for their organization, half of the respondents (50%) said it was a medium priority, or important. This was followed by 34% of respondents indicating that implementation was a high priority, or very important. Twenty-three respondents (14.8%) indicated that implementation was a low priority or not a priority at this time (Figure 4).
Click on image to enlarge. Figure 4. Priorities for Standards Implementation

Insights: Score-Based Likelihood Ratios for Camera Device Identification

INSIGHTS

Score-Based Likelihood Ratios for Camera Device Identification

OVERVIEW

In the developing field of digital image forensics, it is important to be able to identify cameras and other digital devices involved in crimes. However, current camera identification methods fail to quantify the strength of evidence, making it challenging for such evidence to withstand scrutiny in courts. Researchers funded by CSAFE propose using Score-Based Likelihood Ratios to quantify the weight of evidence in digital camera identification.

Lead Researchers

Stephanie Reinders, PhD
Yong Guan, PhD
Danica Ommen, PhD
Jennifer Newman, PhD

Journal

Journal of Forensic Sciences

Publication Date

6 February 2022

Publication Number

IN 126 STAT

Goals

1

Create Score-Based Likelihood Ratios (SLRs) to evaluate the strength of camera identification evidence

2

Compare different SLR models to determine which is the most accurate

The Study

All cameras have small manufacturing imperfections that cause slight variations among pixels in the camera sensor array. These imperfections are known as Photo-Response Non-Uniformities (PRNUs), which create a sort of “camera fingerprint” on images taken with that camera. These PRNUs can be used to identify the device used for a questioned image.

Reinders et al. used a dataset of 4,800 images from a total of 48 known camera devices. They then calculated a similarity score (notated as Δ) between questioned images (Q) and the PRNUs (K) of each camera.

From this, they constructed three different SLRs, each meant to determine the likelihood that a questioned image Q and Person of Interest’s camera’s PRNU K came from the same camera (hypothesis H_p), compared to the likelihood that Q and K came from different cameras (hypothesis H_d).

The three constructed SLR equations

Trace-Anchored SLR: Considers similarity scores between a questioned sample of evidence and samples from the alternative population

Source-Anchored SLR: Considers similarity scores between samples from a specific known source and samples from the alternative population

General Match SLR: Considers similarity scores between samples from randomly selected sources

RESULTS

Of the three questioned methods, the Trace-Anchored SLR had the lowest rate of misleading evidence favoring either the prosecution or the defense with a false positive rate of 2.7% and a false negative rate of 0.5%.
In addition, the Trace-Anchored SLR showed “strong” support for a match in 87.7% of cases where the questioned image came from the Person of Interest’s camera.
By comparison, the General-Match SLR performed the poorest, with false positive rate of 4.7% and a false negative rate of 1.5%. Additionally, the General Match SLR showed “weak” support for a match in 68.6% of cases where the questioned image came from the Person of Interest’s camera.
The Source-Anchored SLR fell in the middle, with a false positive rate of 4.1% and a false negative rate of 0.8%.

Focus on the future

The data used in this study was a closed set, where all images came from the same known 26 devices, and were RAW, center-cropped, auto-exposure, and landscape orientation. Future studies may include an open set, with a larger variety of devices and image types, which may yield different results.

Several researchers have employed an “Inconclusive Zone” that does not result in a definitive match or non-match. This could be included in future studies, and if used in courts, could put further burden of proof on the prosecution and greater benefit of the doubt for the defense.

Additionally, explore relevant publications:

Download

Download the Insight

FULL Study

Access the Full Research Study

More Insights

Insights: Surveying Practicing Firearms Examiners

View Insight

Insights: The Effect of Image Descriptors on the Performance of Classifiers of Footwear Outsole Image Pairs

View Insight

Insights: Score-Based Likelihood Ratios for Camera Device Identification

View Insight

Insights: A Practical Tool for Information Management in Forensic Decisions

View Insight

Insights: Handwriting Identification Using Random Forests and Score-Based Likelihood Ratios

View Insight

Insights: A Practical Tool for Information Management in Forensic Decisions

INSIGHTS

A Practical Tool for Information Management in Forensic Decisions:

Using Linear Sequential Unmasking-Expanded (LSU-E) in Casework

OVERVIEW

While forensic analysts strive to make their findings as accurate and objective as possible, they are often subject to external and internal factors that might bias their decision making. Researchers funded by CSAFE created a practical tool that laboratories can use to implement Linear Sequential Unmasking-Expanded (LSU-E; Dror & Kukucka, 2021)—an information management framework that analysts can use to guide their evaluation of the information available to them. LSU-E can improve decision quality and reduce bias but, until now, laboratories and analysts have
received little concrete guidance to aid implementation efforts.

Lead Researchers

Quigley-McBride, A.
Dror, I.E.
Roy, T.
Garrett, B.L.
Kukucka, J.

Journal

Forensic Science International: Synergy

Publication Date

17 January 2022

Publication Number

IN 125 IMPL

Goals

1

Identify factors that can bias decision-making.

2

Describe how LSU-E can improve forensic decision processes and conclusions.

3

Present a practical worksheet, as well as examples and training materials, to help laboratories incorporate LSU-E into their casework.

TYPES OF COGNITIVE BIAS

Cognitive biases can emerge from a variety of sources, including:

Figure 1. Eight sources of cognitive bias in forensic science (Dror, 2020)

COGNITIVE BIAS IN FORENSIC SCIENCE

As shown in Figure 1, there are many potential sources of information that can influence analysts’ decisions. Of particular concern is suggestive, task-irrelevant contextual information (such as a suspect’s race, sex, or prior criminal record) that can bias analyst’s decisions in inappropriate ways.

In one famous example, FBI latent print analysts concluded with “100 percent certainty” that a print linked to the 2003 Madrid train bombing belonged to a US lawyer, Brandon Mayfield. It transpired that these analysts were all wrong—that was not Mayfield’s print. Mayfield was Muslim, which might have biased the analysts given the strong, widespread attitudes towards Muslims post 9/11. Also, Mayfield was on the FBI’s “watch list” because he provided legal representation to someone accused of terrorist activities. Combined, these facts led to confirmation bias effects in the analysts’ evaluations and conclusions about Mayfield’s fingerprints.

LSU-E AND INFORMATION
MANAGEMENT

LSU-E is an approach information management which prioritizes case information based on three main criteria:

Biasing power:

How strongly the information might dispose an analyst to a particular conclusion.

Objectivity:

The extent to which the information might be interpreted to have different “meanings” from one analyst to another.

Relevance:

the degree to which the information is essential to the analytic task itself.

IMPLEMENTING LSU-E IN
FORENSICS

Quigley-McBride et al. have created a practical worksheet for laboratories to use when assessing new information.

1

First, the user specifies the information in question and its source

2

Second, they consider the three LSU-E criteria, and rate the information on a scale of 1-5 for each criterion

3

Finally, they describe strategies to minimize any adverse effects the information may have on the decision-making process

Focus on the future

Ideally, LSU-E procedures would be applied before the information reaches the analyst. That said, it is still effective when used at any point in the analyst’s workflow and can help analysts become aware of information that can inappropriately influence their work.

In addition to benefits for analysts, implementing LSU-E could help jurors evaluate the reliability of forensic expert testimony. This would not only encourage healthy skepticism among jurors, but could bolster an expert’s credibility by providing documentation of methods used to evaluate and mitigate potential biases in their decisions.

Additional relevant publications and presentations:

Download

Download the Insight

FULL Study

Access the Full Research Study

Month: February 2022

From Results from a Black-Box Study for Digital Forensic Examiners, page 33:

From page 10 of the OSAC Registry Implementation Survey: 2021 Report:

INSIGHTS

Score-Based Likelihood Ratios for Camera Device Identification

OVERVIEW

Lead Researchers

Journal

Publication Date

Publication Number

Goals

1

2

The Study

RESULTS

Focus on the future

Additionally, explore relevant publications:

Download

FULL Study

More Insights

INSIGHTS

A Practical Tool for Information Management in Forensic Decisions:

OVERVIEW

Lead Researchers

Journal

Publication Date

Publication Number

Goals

1

2

3

TYPES OF COGNITIVE BIAS

COGNITIVE BIAS IN FORENSIC SCIENCE

LSU-E AND INFORMATIONMANAGEMENT

Biasing power:

Objectivity:

Relevance:

IMPLEMENTING LSU-E INFORENSICS

1

2

3

Focus on the future

Additional relevant publications and presentations:

Download

FULL Study

More Insights

Partially Funded by

LSU-E AND INFORMATION
MANAGEMENT

IMPLEMENTING LSU-E IN
FORENSICS