mlsanityData to 14 Oct 2026

Measuring Coding Challenge Competence With APPS

arXiv 2105.09938 ↗First posted 20 May 2021cs.SE cs.CL cs.LGcited by 902

Not on the map. Why?

Cited by 138 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements.

Key facts. Cited by 902 papers in the map (data through 14 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 20 May 2021.

Builds on

26 of 50 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2018Datasheets for Datasetscited by 1,386
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  11. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  12. 2020Aligning AI With Shared Human Valuescited by 723
14 more
  1. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  2. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  3. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  4. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
  5. 2020Unsupervised Translation of Programming Languagescited by 263
  6. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  7. 2019SPoC: Search-based Pseudocode to Codecited by 176
  8. 2018Mapping Language to Code in Programmatic Contextcited by 104
  9. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 94
  10. 2016Latent Predictor Networks for Code Generationcited by 73
  11. 2015Program Synthesis using Natural Languagecited by 41
  12. 2017Making Neural Programming Architectures Generalize via Recursioncited by 35
  13. 2018NAPS: Natural Program Synthesis Datasetcited by 14
  14. 2019SyGuS-Comp 2018: Results and Analysiscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 45; 2023: 111; 2024: 210; 2025: 259; 2026: 267.

902 papers.

Sort:

By area:

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  9. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  10. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  11. 2023Teaching Large Language Models to Self-Debugcited by 988
  12. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  4. 2024Large Language Models: A Surveycited by 525
  5. 2025Process Reinforcement through Implicit Rewardscited by 504
  6. 2025Humanity's Last Examcited by 491
  7. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  8. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  9. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  10. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  11. 2022CodeT: Code Generation with Generated Testscited by 384
  12. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  13. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  14. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  15. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  16. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  17. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  18. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  19. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  20. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  21. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  22. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  23. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  24. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  25. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  26. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  27. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  28. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  29. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  30. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  31. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  32. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  33. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  34. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  35. 2025Survey on Evaluation of LLM-based Agentscited by 204
  36. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  37. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
  38. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197

50 most cited of 902 All 902 (JSON)