mlsanityData to 7 Oct 2026

Execution-Based Evaluation for Open-Domain Code Generation

arXiv 2212.10481 ↗First posted 20 Dec 2022cs.SE cs.AI cs.CLcited by 89

Not on the map. Why?

Abstract (arXiv)

To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human-written test cases for execution. Our NL-Code pairs are harvested from StackOverflow forums to encourage natural and practical coding queries. Moreover, ODEX supports four natural languages as intents, in English, Spanish, Japanese, and Russian. ODEX unveils intriguing behavioral differences among top-performing code language models (LM). While CODEX achieves better overall results, CODEGEN improves effectively via scaling -- CODEGEN 6.1B performs comparably with CODEX 12B. Both models show substantial gaps between open and closed domains, but CODEGEN gaps tend to decrease with model size while CODEX gaps increase. We release ODEX to facilitate research into open-domain problems for the code generation community.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

28 of 40 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  9. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  11. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  12. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
16 more
  1. 2022CodeT: Code Generation with Generated Testscited by 369
  2. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  3. 2016Language to Logical Form with Neural Attentioncited by 158
  4. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  5. 2018Mapping Language to Code in Programmatic Contextcited by 104
  6. 2018TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generationcited by 88
  7. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  8. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  9. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  10. 2022Pynguin: Automated Unit Test Generation for Pythoncited by 49
  11. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  12. 2022Training and Evaluating a Jupyter Notebook Data Science Assistantcited by 31
  13. 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 31
  14. 2020Automated Unit Test Generation for Pythoncited by 18
  15. 2019An Empirical Study of C++ Vulnerabilities in Crowd-Sourced Code Examplescited by 7
  16. 2021Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questionscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 43; 2025: 17; 2026: 13.

89 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2023StarCoder: may the source be with you!cited by 790
  5. 2024A Survey on Large Language Models for Code Generationcited by 646
  6. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  7. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  8. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  9. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  10. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  11. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  12. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
38 more
  1. 2025Survey on Evaluation of LLM-based Agentscited by 190
  2. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  3. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  4. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  5. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  6. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 89
  7. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  8. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  9. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  10. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 53
  11. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  12. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  13. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 46
  14. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  15. 2024Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Studycited by 40
  16. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 38
  17. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 38
  18. 2025MultiKernelBench: A Multi-Platform Benchmark for Kernel Generationcited by 35
  19. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  20. 2024SWE-bench-java: A GitHub Issue Resolving Benchmark for Javacited by 34
  21. 2024ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systemscited by 30
  22. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  23. 2023An In-depth Look at Gemini's Language Abilitiescited by 28
  24. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  25. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 25
  26. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  27. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  28. 2024Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Taskscited by 24
  29. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 24
  30. 2024CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarkscited by 23
  31. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22
  32. 2024EVOR: Evolving Retrieval for Code Generationcited by 22
  33. 2024On the Impacts of Contexts on Repository-Level Code Generationcited by 21
  34. 2024DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generationcited by 21
  35. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20
  36. 2025A Study of LLMs' Preferences for Libraries and Programming Languagescited by 20
  37. 2025Towards Enforcing Company Policy Adherence in Agentic Workflowscited by 17
  38. 2024RePair: Automated Program Repair with Process-based Feedbackcited by 16

50 most cited of 89 All 89 (JSON)