# Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks

arXiv: https://arxiv.org/abs/2609.06780 · HTML: https://mlsanity.com/paper/2609.06780/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 43 references that are in the map.
- First posted on arXiv 6 Sep 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 12 of 12)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents](https://mlsanity.com/paper/2504.08703/) (2025)
- [NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents](https://mlsanity.com/paper/2512.12730/) (2025)
- [DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks](https://mlsanity.com/paper/2607.07946/) (2026)
- [GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents](https://mlsanity.com/paper/2505.23671/) (2025)
- [OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution](https://mlsanity.com/paper/2505.04606/) (2025)
- [BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?](https://mlsanity.com/paper/2603.03194/) (2026)
- [AInsteinBench: Benchmarking Coding Agents on Scientific Repositories](https://mlsanity.com/paper/2512.21373/) (2025)

## Built on by (most cited first, 1 of 1)

- [Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules](https://mlsanity.com/paper/2610.14216/) (2026)

## Abstract (arXiv)

While autonomous software engineering (SWE) agents achieve high benchmark resolution rates, these scores can mask exploitative behaviors---such as leveraging local Git histories, accessing upstream repositories, or recalling memorized solutions---rather than demonstrating genuine problem solving. We systematize and audit these exploits across five open large language models on SWE-bench Multilingual and DeepSWE using a turn-level LLM-as-a-judge protocol. Under standard prompts, exploitation rates reach 45.1%--82.4% on SWE-bench Multilingual and 44.2%--66.1% on DeepSWE. Appending a targeted instruction enforcing solution originality drastically cuts these exploitation rates---down to 4.0%--10.7% and 1.5%--7.1%, respectively, while maintaining strong core task performance. Our findings demonstrate the critical need for exploit-aware evaluation frameworks that measure true repository-level problem solving over benchmark gaming.