mlsanityData to 14 Oct 2026

InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

arXiv 2306.14898 ↗First posted 26 Jun 2023cs.CL cs.LG cs.SEcited by 238

Not on the map. Why?

Cited by 129 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while enabling the development of new methods for interactive code generation. We use InterCode to create three interactive code environments with Bash, SQL, and Python as action spaces, leveraging data from the static NL2Bash, Spider, and MBPP datasets. We demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results showcase the benefits of interactive code generation and demonstrate that InterCode can serve as a challenging benchmark for advancing code understanding and generation capabilities. InterCode is designed to be easily extensible and can even be used to create new tasks such as Capture the Flag, a popular coding puzzle that is inherently multi-step and involves multiple programming languages. Project site with code and data: https://intercode-benchmark.github.io

Key facts. Cited by 238 papers in the map (data through 14 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

45 of 59 references.

References · 45

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  12. 2023Teaching Large Language Models to Self-Debugcited by 988
33 more
  1. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  2. 2021Measuring Coding Challenge Competence With APPScited by 902
  3. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  4. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  5. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  6. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  7. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  8. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  9. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  10. 2022CodeT: Code Generation with Generated Testscited by 384
  11. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  12. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  13. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
  14. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 185
  15. 2016Language to Logical Form with Neural Attentioncited by 158
  16. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 144
  17. 2022Natural Language to Code Translation with Executioncited by 124
  18. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  19. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  20. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 102
  21. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 98
  22. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 93
  23. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  24. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  25. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 84
  26. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  27. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 76
  28. 2022Interactive Code Generation via Test-Driven User-Intent Formalizationcited by 50
  29. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  30. 2022Compilable Neural Code Generation with Compiler Feedbackcited by 47
  31. 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 32
  32. 2023LeTI: Learning to Generate from Textual Interactionscited by 25
  33. 2021NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commandscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 50; 2025: 72; 2026: 100.

238 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  5. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  6. 2023Cognitive Architectures for Language Agentscited by 429
  7. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 315
  8. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  9. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  10. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  11. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 276
  12. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
38 more
  1. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
  2. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  3. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 177
  4. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  5. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 140
  6. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 137
  7. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  8. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 119
  9. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  10. 2024DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agentscited by 100
  11. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 93
  12. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 78
  13. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 73
  14. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 72
  15. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  16. 2024StateFlow: Enhancing LLM Task-Solving through State-Driven Workflowscited by 65
  17. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  18. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  19. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 58
  20. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 56
  21. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  22. 2026Endless Terminals: Scaling RL Environments for Terminal Agentscited by 54
  23. 2023EcoAssistant: Using LLM Assistant More Affordably and Accuratelycited by 49
  24. 2024When is Tree Search Useful for LLM Planning? It Depends on the Discriminatorcited by 47
  25. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  26. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  27. 2025DABstep: Data Agent Benchmark for Multi-step Reasoningcited by 46
  28. 2024Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Studycited by 44
  29. 2025BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systemscited by 40
  30. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 39
  31. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
  32. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solvingcited by 38
  33. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 38
  34. 2024Benchmarking Agentic Workflow Generationcited by 37
  35. 2025ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Explorationcited by 37
  36. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  37. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  38. 2026SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolvingcited by 31

50 most cited of 238 All 238 (JSON)