# CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions

arXiv: https://arxiv.org/abs/2602.20213 · HTML: https://mlsanity.com/paper/2602.20213/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 23 Feb 2026.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [CodeContests+: High-Quality Test Case Generation for Competitive Programming](https://mlsanity.com/paper/2506.05817/) (2025)
- [HardTests: Synthesizing High-Quality Test Cases for LLM Coding](https://mlsanity.com/paper/2505.24098/) (2025)
- [LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs](https://mlsanity.com/paper/2404.10304/) (2024)
- [From Reasoning to Code: GRPO Optimization for Underrepresented Languages](https://mlsanity.com/paper/2506.11027/) (2025)
- [Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems](https://mlsanity.com/paper/2506.06821/) (2025)
- [VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection](https://mlsanity.com/paper/2509.11523/) (2025)
- [From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level](https://mlsanity.com/paper/2601.03731/) (2026)
- [TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation](https://mlsanity.com/paper/2508.11468/) (2025)
- [Codehacks: A Dataset of Adversarial Tests for Competitive Programming Problems Obtained from Codeforces](https://mlsanity.com/paper/2503.23466/) (2025)

## Built on by (most cited first, 8 of 8)

- [Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning](https://mlsanity.com/paper/2604.27998/) (2026)
- [SWE-AGILE: A Software Agent Framework for Efficiently Managing Dynamic Reasoning Context](https://mlsanity.com/paper/2604.11716/) (2026)
- [When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR](https://mlsanity.com/paper/2607.11022/) (2026)
- [VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation](https://mlsanity.com/paper/2605.08553/) (2026)
- [Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution](https://mlsanity.com/paper/2605.15301/) (2026)
- [Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch](https://mlsanity.com/paper/2608.01715/) (2026)
- [GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing](https://mlsanity.com/paper/2610.02952/) (2026)
- [Beyond the Verdict: Agentic Code Judging with Verifiable Feedback](https://mlsanity.com/paper/2610.18760/) (2026)

## Abstract (arXiv)

The evaluation of Large Language Models (LLMs) for code generation relies heavily on the quality and robustness of test cases. However, existing benchmarks often lack coverage for subtle corner cases, allowing incorrect solutions to pass. To bridge this gap, we propose CodeHacker, an automated agent framework dedicated to generating targeted adversarial test cases that expose latent vulnerabilities in program submissions. Mimicking the hack mechanism in competitive programming, CodeHacker employs a multi-strategy approach, including stress testing, anti-hash attacks, and logic-specific targeting to break specific code submissions. To ensure the validity and reliability of these attacks, we introduce a Calibration Phase, where the agent iteratively refines its own Validator and Checker via self-generated adversarial probes before evaluating contestant code.Experiments demonstrate that CodeHacker significantly improves the True Negative Rate (TNR) of existing datasets, effectively filtering out incorrect solutions that were previously accepted. Furthermore, generated adversarial cases prove to be superior training data, boosting the performance of RL-trained models on benchmarks like LiveCodeBench.