# Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence

arXiv: https://arxiv.org/abs/2609.19965 · HTML: https://mlsanity.com/paper/2609.19965/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 41 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Large Language Models Assume People are More Rational than We Really are](https://mlsanity.com/paper/2406.17055/) (2024)
- [Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration](https://mlsanity.com/paper/2310.09241/) (2023)
- [Presumed Cultural Identity: How Names Shape LLM Responses](https://mlsanity.com/paper/2502.11995/) (2025)
- [Unilaw-R1: A Large Language Model for Legal Reasoning with Reinforcement Learning and Iterative Inference](https://mlsanity.com/paper/2510.10072/) (2025)
- [Guiding LLM Decision-Making with Fairness Reward Models](https://mlsanity.com/paper/2507.11344/) (2025)
- [SGIC: A Self-Guided Iterative Calibration Framework for RAG](https://mlsanity.com/paper/2506.16172/) (2025)
- [CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions](https://mlsanity.com/paper/2501.00097/) (2024)
- [WCLD: Curated Large Dataset of Criminal Cases from Wisconsin Circuit Courts](https://mlsanity.com/paper/2310.18724/) (2023)
- [A Two-Stage Prediction-Aware Contrastive Learning Framework for Multi-Intent NLU](https://mlsanity.com/paper/2405.02925/) (2024)
- [LAPIS: Language Model-Augmented Police Investigation System](https://mlsanity.com/paper/2407.20248/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly applied to legal and criminal justice tasks, yet existing work focuses almost exclusively on post-arrest scenarios where the suspect's identity is already known, leaving the critical pre-arrest challenge of inferring suspect characteristics from incomplete evidence largely unexplored. To fill this gap, we introduce the Profiling, Investigation, and Judgment (PIJ), comprising 2,500 real homicide cases from five countries. PIJ evaluates LLMs across three tasks that span the entire criminal investigation pipeline: criminal profiling, which requires abductive reasoning to infer suspect attributes from fragmentary scene evidence, crime process reconstruction, which tests structured information extraction, and sentence prediction, which demands legal deductive reasoning. We evaluate 9 powerful LLMs and find that performance degrades systematically as tasks shift from explicit fact extraction to implicit reasoning over unknown suspect profiles. Categories requiring inferential reasoning, such as motivation and victim-offender relationships, remain the primary bottlenecks. Further analysis reveals substantial gaps between LLMs and human experts, along with pervasive biases in gender, age, and motive attribution. Our findings indicate that pre-arrest inference from incomplete evidence remains an open challenge.