# The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason

arXiv: https://arxiv.org/abs/2506.12286 · HTML: https://mlsanity.com/paper/2506.12286/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 14 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 15 of 15)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation](https://mlsanity.com/paper/2309.10677/) (2023)
- [RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code](https://mlsanity.com/paper/2503.07832/) (2025)
- [Are Large Language Models Memorizing Bug Benchmarks?](https://mlsanity.com/paper/2411.13323/) (2024)
- [Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting](https://mlsanity.com/paper/2503.02296/) (2025)
- [TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models](https://mlsanity.com/paper/2407.21227/) (2024)

## Built on by (most cited first, 30 of 49)

- [DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks](https://mlsanity.com/paper/2607.07946/) (2026)
- [Does SWE-Bench-Verified Test Agent Ability or Model Memory?](https://mlsanity.com/paper/2512.10218/) (2025)
- [Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation](https://mlsanity.com/paper/2510.08996/) (2025)
- [The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering](https://mlsanity.com/paper/2507.15003/) (2025)
- [PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents](https://mlsanity.com/paper/2608.04003/) (2026)
- [Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems](https://mlsanity.com/paper/2506.17208/) (2025)
- [Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback](https://mlsanity.com/paper/2507.18755/) (2025)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents](https://mlsanity.com/paper/2606.05391/) (2026)
- [SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle](https://mlsanity.com/paper/2605.13139/) (2026)
- [Auditing Reward Hackability in Code RL Training Environments](https://mlsanity.com/paper/2606.16062/) (2026)
- [What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents](https://mlsanity.com/paper/2607.06184/) (2026)
- [SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?](https://mlsanity.com/paper/2608.23564/) (2026)
- [Gistify! Codebase-Level Understanding via Runtime Execution](https://mlsanity.com/paper/2510.26790/) (2025)
- [From Plan to Action: How Well Do Agents Follow the Plan?](https://mlsanity.com/paper/2604.12147/) (2026)
- [SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion](https://mlsanity.com/paper/2510.17925/) (2025)
- [RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations](https://mlsanity.com/paper/2605.26177/) (2026)
- [Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering](https://mlsanity.com/paper/2606.17799/) (2026)
- [The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior](https://mlsanity.com/paper/2608.11386/) (2026)
- [LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering](https://mlsanity.com/paper/2608.28281/) (2026)
- [Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoning](https://mlsanity.com/paper/2508.10057/) (2025)
- [Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis](https://mlsanity.com/paper/2603.21454/) (2026)
- [Code Isn't Memory: A Structural Codebase Index Inside a Coding Agent](https://mlsanity.com/paper/2606.22417/) (2026)
- [SHERLOC: Structured Diagnostic Localization for Code Repair Agents](https://mlsanity.com/paper/2606.24820/) (2026)
- [Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study](https://mlsanity.com/paper/2607.02436/) (2026)
- [Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Models](https://mlsanity.com/paper/2511.00710/) (2025)
- [On the Limits of Innate Planning in Large Language Models](https://mlsanity.com/paper/2511.21591/) (2025)
- [Debug2Fix: Can Interactive Debugging Help Coding Agents Fix More Bugs?](https://mlsanity.com/paper/2602.18571/) (2026)
- [Resolving Java Code Repository Issues with iSWE Agent](https://mlsanity.com/paper/2603.11356/) (2026)
- [The Kitchen Loop: User-Spec-Driven Development for a Self-Evolving Codebase](https://mlsanity.com/paper/2603.25697/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become increasingly capable and widely adopted, benchmarks play a central role in assessing their practical utility. For example, SWE-Bench Verified has emerged as a critical benchmark for evaluating LLMs' software engineering abilities, particularly their aptitude for resolving real-world GitHub issues. Recent LLMs show impressive performance on SWE-Bench, leading to optimism about their capacity for complex coding tasks. However, current evaluation protocols may overstate these models' true capabilities. It is crucial to distinguish LLMs' generalizable problem-solving ability and other learned artifacts. In this work, we introduce two diagnostic tasks: file path identification from issue descriptions alone and ground truth function reproduction with only the current file context and issue description to probe models' underlying knowledge. We present empirical evidence that performance gains on SWE-Bench-Verified may be partially driven by memorization rather than genuine problem-solving. We show that state-of-the-art models achieve up to 76% accuracy in identifying buggy file paths using only issue descriptions, without access to repository structure. This performance is merely up to 53% on tasks from repositories not included in SWE-Bench, pointing to possible data contamination or memorization. Similar patterns are also observed for the function reproduction task, where the verbatim similarity is much higher on SWE-Bench Verified than on other similar coding benchmarks (up to 35% consecutive 5-gram accuracy on SWE-Bench Verified and Full, but only up to 18% for tasks in other benchmarks). These findings raise concerns about the validity of existing results and underscore the need for more robust, contamination-resistant benchmarks to reliably evaluate LLMs' coding abilities.