Codehacks: A Dataset of Adversarial Tests for Competitive Programming Problems Obtained from Codeforces
Not on the map. Why?
Abstract (arXiv)
Software is used in critical applications in our day-to-day life and it is important to ensure its correctness. One popular approach to assess correctness is to evaluate software on tests. If a test fails, it indicates a fault in the software under test; if all tests pass correctly, one may assume that the software is correct. However, the reliability of these results depends on the test suite considered, and there is a risk of false negatives (i.e. software that passes all available tests but contains bugs because some cases are not tested). Therefore, it is important to consider error-inducing test cases when evaluating software. To support data-driven creation of such a test-suite, which is especially of interest for testing software synthesized from large language models, we curate a dataset (Codehacks) of programming problems together with corresponding error-inducing test cases (i.e., "hacks"). This dataset is collected from the wild, in particular, from the Codeforces online judge platform. The dataset comprises 288,617 hacks for 5,578 programming problems, each with a natural language description, as well as the source code for 2,196 submitted solutions to these problems that can be broken with their corresponding hacks. Keywords: competitive programming, language model, dataset
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 30 Mar 2025.
Builds on
15 of 23 references.
References · 15
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2022Competition-Level Code Generation with AlphaCodecited by 1,462
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2022CodeT: Code Generation with Generated Testscited by 384
- 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
- 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 181
- 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
- 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
- 2023Fully Autonomous Programming with Large Language Modelscited by 31
- 2022Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Codecited by 21
Built on by
Citing papers per year: 2026: 2.
2 papers.