# Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"

arXiv: https://arxiv.org/abs/2502.06898 · HTML: https://mlsanity.com/paper/2502.06898/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Make Your LLM Fully Utilize the Context](https://mlsanity.com/paper/2404.16811/) (2024)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks](https://mlsanity.com/paper/2312.12575/) (2023)
- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [Prompt-Enhanced Software Vulnerability Detection Using ChatGPT](https://mlsanity.com/paper/2308.12697/) (2023)
- [Large Language Model for Vulnerability Detection: Emerging Results and Future Directions](https://mlsanity.com/paper/2401.15468/) (2024)
- [ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We?](https://mlsanity.com/paper/2310.09810/) (2023)
- [When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?](https://mlsanity.com/paper/2309.05520/) (2023)
- [Transformer-Based Language Models for Software Vulnerability Detection](https://mlsanity.com/paper/2204.03214/) (2022)
- [ReposVul: A Repository-Level High-Quality Vulnerability Dataset](https://mlsanity.com/paper/2401.13169/) (2024)
- [Evaluation of ChatGPT Model for Vulnerability Detection](https://mlsanity.com/paper/2304.07232/) (2023)
- [Giraffe: Adventures in Expanding Context Lengths in LLMs](https://mlsanity.com/paper/2308.10882/) (2023)
- [LLM In-Context Recall is Prompt Dependent](https://mlsanity.com/paper/2404.08865/) (2024)
- [Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answers](https://mlsanity.com/paper/2403.15600/) (2024)
- [A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages](https://mlsanity.com/paper/2308.04477/) (2023)
- [Comparing large language models and human programmers for generating programming code](https://mlsanity.com/paper/2403.00894/) (2024)

## Built on by (most cited first, 5 of 5)

- [A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models](https://mlsanity.com/paper/2507.22659/) (2025)
- [From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localization](https://mlsanity.com/paper/2510.02389/) (2025)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [Security and Privacy in Agentic AI: Grand Challenges and Future Directions](https://mlsanity.com/paper/2607.06608/) (2026)

## Abstract (arXiv)

Recent advancements in artificial intelligence have enabled processing of larger inputs, leading everyday software developers to increasingly rely on chat-based large language models (LLMs) like GPT-3.5 and GPT-4 to detect vulnerabilities across entire files, not just within functions. This new development practice requires researchers to urgently investigate whether commonly used LLMs can effectively analyze large file-sized inputs, in order to provide timely insights for software developers and engineers about the pros and cons of this emerging technological trend. Hence, the goal of this paper is to evaluate the effectiveness of several state-of-the-art chat-based LLMs, including the GPT models, in detecting in-file vulnerabilities. We conducted a costly investigation into how the performance of LLMs varies based on vulnerability type, input size, and vulnerability location within the file. To give enough statistical power to our study, we could only focus on the three most common (as well as dangerous) vulnerabilities: XSS, SQL injection, and path traversal. Our findings indicate that the effectiveness of LLMs in detecting these vulnerabilities is strongly influenced by both the location of the vulnerability and the overall size of the input. Specifically, regardless of the vulnerability type, LLMs tend to significantly (p < .05) underperform when detecting vulnerabilities located toward the end of larger files, a pattern we call the 'lost-in-the-end' effect. Finally, to further support software developers and practitioners, we also explored the optimal input size for these LLMs and presented a simple strategy for identifying it, which can be applied to other models and vulnerability types. Eventually, we show how adjusting the input size can lead to significant improvements in LLM-based vulnerability detection, with an average recall increase of over 37% across all models.