# Chasing Shadows: Pitfalls in LLM Security Research

arXiv: https://arxiv.org/abs/2512.09549 · HTML: https://mlsanity.com/paper/2512.09549/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2025.

## Built on by (most cited first, 13 of 13)

- [The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility](https://mlsanity.com/paper/2605.19537/) (2026)
- [Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesis](https://mlsanity.com/paper/2605.00314/) (2026)
- [SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity](https://mlsanity.com/paper/2602.08690/) (2026)
- [Towards Reliable Local Security Agents: Verifiable Post-Training for Linux Privilege Escalation](https://mlsanity.com/paper/2603.17673/) (2026)
- [Probing Memorization of Tabular In-Context Learning](https://mlsanity.com/paper/2606.31208/) (2026)
- [From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models](https://mlsanity.com/paper/2601.22946/) (2026)
- [A Comprehensive Evaluation of Code Language Models for Security Patch Detection](https://mlsanity.com/paper/2605.13138/) (2026)
- [Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks](https://mlsanity.com/paper/2605.23243/) (2026)
- [SoK: AI-Augmented Binary Reversing](https://mlsanity.com/paper/2606.17398/) (2026)
- [The Half-Lives of Generative-AI Evidence: A 40-Record Audit, a Claim-Currency Framework, and a Reflexive Case of Frontier-Model-Assisted Research](https://mlsanity.com/paper/2607.24032/) (2026)
- [Misinformation Without Triggers: From Factual Answers to Downstream Decisions](https://mlsanity.com/paper/2610.02886/) (2026)
- [Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents](https://mlsanity.com/paper/2610.03448/) (2026)
- [Learning from Failures: A Failure-Driven Prompt Refinement for LLM-Based Vulnerability Analysis](https://mlsanity.com/paper/2610.08405/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly prevalent in security research. Their unique characteristics, however, introduce challenges that undermine established paradigms of reproducibility, rigor, and evaluation. Prior work has identified common pitfalls in traditional machine learning research, but these studies predate the advent of LLMs. In this paper, we identify nine common pitfalls that have become (more) relevant with the emergence of LLMs and that can compromise the validity of research involving them. These pitfalls span the entire computation process, from data collection, pre-training, and fine-tuning to prompting and evaluation. We assess the prevalence of these pitfalls across all 72 peer-reviewed papers published at leading Security and Software Engineering venues between 2023 and 2024. We find that every paper contains at least one pitfall, and each pitfall appears in multiple papers. Yet only 15.7% of the present pitfalls were explicitly discussed, suggesting that the majority remain unrecognized. To understand their practical impact, we conduct four empirical case studies showing how individual pitfalls can mislead evaluation, inflate performance, or impair reproducibility. Based on our findings, we offer actionable guidelines to support the community in future work.