# DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Response

arXiv: https://arxiv.org/abs/2505.19973 · HTML: https://mlsanity.com/paper/2505.19973/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation](https://mlsanity.com/paper/2402.11443/) (2024)
- [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://mlsanity.com/paper/2406.07599/) (2024)
- [CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge](https://mlsanity.com/paper/2402.07688/) (2024)
- [Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devices](https://mlsanity.com/paper/2306.14263/) (2023)
- [Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities](https://mlsanity.com/paper/2405.12750/) (2024)
- [ChatGPT for Digital Forensic Investigation: The Good, The Bad, and The Unknown](https://mlsanity.com/paper/2307.10195/) (2023)
- [Digital Forensics in the Age of Large Language Models](https://mlsanity.com/paper/2504.02963/) (2025)
- [ChatGPT, Llama, can you write my report? An experiment on assisted digital forensics reports written using (Local) Large Language Models](https://mlsanity.com/paper/2312.14607/) (2023)
- [GenDFIR: Advancing Cyber Incident Timeline Analysis Through Retrieval Augmented Generation and Large Language Models](https://mlsanity.com/paper/2409.02572/) (2024)
- [AVScan2Vec: Feature Learning on Antivirus Scan Data for Production-Scale Malware Corpora](https://mlsanity.com/paper/2306.06228/) (2023)
- [Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence](https://mlsanity.com/paper/2410.15490/) (2024)
- [Collaborative Feature Maps of Networks and Hosts for AI-driven Intrusion Detection](https://mlsanity.com/paper/2208.05085/) (2022)
- [Towards a standardized methodology and dataset for evaluating LLM-based digital forensic timeline analysis](https://mlsanity.com/paper/2505.03100/) (2025)

## Built on by (most cited first, 4 of 4)

- [AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor](https://mlsanity.com/paper/2601.05752/) (2026)
- [SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents](https://mlsanity.com/paper/2604.12040/) (2026)
- [TxRay: Agentic Postmortem of Live Blockchain Attacks](https://mlsanity.com/paper/2602.01317/) (2026)
- [Generation of Programmatic Rules for Document Forgery Detection Using Large Language Models](https://mlsanity.com/paper/2512.19228/) (2025)

## Abstract (arXiv)

Digital Forensics and Incident Response (DFIR) involves analyzing digital evidence to support legal investigations. Large Language Models (LLMs) offer new opportunities in DFIR tasks such as log analysis and memory forensics, but their susceptibility to errors and hallucinations raises concerns in high-stakes contexts. Despite growing interest, there is no comprehensive benchmark to evaluate LLMs across both theoretical and practical DFIR domains. To address this gap, we present DFIR-Metric, a benchmark with three components: (1) Knowledge Assessment: a set of 700 expert-reviewed multiple-choice questions sourced from industry-standard certifications and official documentation; (2) Realistic Forensic Challenges: 150 CTF-style tasks testing multi-step reasoning and evidence correlation; and (3) Practical Analysis: 500 disk and memory forensics cases from the NIST Computer Forensics Tool Testing Program (CFTT). We evaluated 14 LLMs using DFIR-Metric, analyzing both their accuracy and consistency across trials. We also introduce a new metric, the Task Understanding Score (TUS), designed to more effectively evaluate models in scenarios where they achieve near-zero accuracy. This benchmark offers a rigorous, reproducible foundation for advancing AI in digital forensics. All scripts, artifacts, and results are available on the project website at https://github.com/DFIR-Metric.