mlsanityData to 7 Oct 2026

If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents

arXiv 2401.00812 ↗First posted 1 Jan 2024cs.CLcited by 77

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The prominent large language models (LLMs) of today differ from past language models not only in size, but also in the fact that they are trained on a combination of natural language and formal language (code). As a medium between humans and computers, code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity. In this survey, we present an overview of the various benefits of integrating code into LLMs' training data. Specifically, beyond enhancing LLMs in code generation, we observe that these unique properties of code help (i) unlock the reasoning ability of LLMs, enabling their applications to a range of more complex natural language tasks; (ii) steer LLMs to produce structured and precise intermediate steps, which can then be connected to external execution ends through function calls; and (iii) take advantage of code compilation and execution environment, which also provides diverse feedback for model improvement. In addition, we trace how these profound capabilities of LLMs, brought by code, have led to their emergence as intelligent agents (IAs) in situations where the ability to understand instructions, decompose goals, plan and execute actions, and refine from feedback are crucial to their success on downstream tasks. Finally, we present several key challenges and future directions of empowering LLMs with code.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 110 of its 158 references that are in the map. First posted on arXiv 1 Jan 2024.

Builds on

110 of 158 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Lemur: Harmonizing Natural Language and Code for Language Agents10 Oct
  2. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration3 Oct
  3. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agents23 Sep
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  5. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  6. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug
  7. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents5 Jun

Other references · 103

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
38 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  2. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  12. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  13. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  14. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  15. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  16. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  17. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  18. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  19. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  20. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  21. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  22. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  23. 2022CodeT: Code Generation with Generated Testscited by 369
  24. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  25. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  26. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  27. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  28. 2023Large Language Models as Tool Makerscited by 293
  29. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  30. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  31. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  32. 2023Faithful Chain-of-Thought Reasoningcited by 285
  33. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  34. 2022Autoformalization with Large Language Modelscited by 252
  35. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  36. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  37. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  38. 2022Binding Language Models in Symbolic Languagescited by 224

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 46; 2025: 24; 2026: 7.

77 papers.

Sort:

By area:

  1. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  2. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  3. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 110
  4. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  5. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  6. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  7. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agentscited by 61
  8. 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
  9. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agentscited by 56
  10. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  11. 2025Self-Challenging Language Model Agentscited by 56
  12. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
38 more
  1. 2024MASAI: Modular Architecture for Software-engineering AI Agentscited by 49
  2. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  3. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 41
  4. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 39
  5. 2024Benchmarking Agentic Workflow Generationcited by 36
  6. 2024Code Pretraining Improves Entity Tracking Abilities of Language Modelscited by 31
  7. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  8. 2024Prompting Large Language Models for Zero-Shot Clinical Prediction with Structured Longitudinal Electronic Health Record Datacited by 22
  9. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  10. 2024Divide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generationcited by 19
  11. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 18
  12. 2025Test-Time Adaptation for LLM Agents via Environment Interactioncited by 17
  13. 2024EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundariescited by 16
  14. 2024Stable Code Technical Reportcited by 16
  15. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  16. 2024Unlocking the Power of Large Language Models for Entity Alignmentcited by 15
  17. 2024Abstraction-of-Thought Makes Language Models Better Reasonerscited by 11
  18. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  19. 2024CodeTaxo: Enhancing Taxonomy Expansion with Limited Examples via Code Language Promptscited by 10
  20. 2024PropaInsight: Toward Deeper Understanding of Propaganda in Terms of Techniques, Appeals, and Intentcited by 9
  21. 2024Python is Not Always the Best Choice: Embracing Multilingual Program of Thoughtscited by 8
  22. 2024Benchmarking ChatGPT on Algorithmic Reasoningcited by 8
  23. 2024Practical Considerations for Agentic LLM Systemscited by 8
  24. 2025ToolCoder: A Systematic Code-Empowered Tool Learning Framework for Large Language Modelscited by 8
  25. 2026ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognitioncited by 8
  26. 2024Are LLMs classical or nonmonotonic reasoners? Lessons from genericscited by 7
  27. 2024Rx Strategist: Prescription Verification using LLM Agents Systemcited by 7
  28. 2024TextLap: Customizing Language Models for Text-to-Layout Planningcited by 7
  29. 2025Talk to Your Slides: High-Efficiency Slide Editing via Language-Driven Structured Data Manipulationcited by 7
  30. 2025SimuGen: Multi-modal Agentic Framework for Constructing Block Diagram-Based Simulation Modelscited by 7
  31. 2024Levels of AI Agents: from Rules to Large Language Modelscited by 6
  32. 2024AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Datacited by 6
  33. 2024CodeGraph: Enhancing Graph Reasoning of LLMs with Codecited by 6
  34. 2024CursorCore: Assist Programming through Aligning Anythingcited by 6
  35. 2025Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?cited by 6
  36. 2025Safe, Untrusted, "Proof-Carrying" AI Agents: toward the agentic lakehousecited by 6
  37. 2025Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memorycited by 6
  38. 2024IEPile: Unearthing Large-Scale Schema-Based Information Extraction Corpuscited by 5

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence
  2. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  3. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  4. 2026LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review
  5. 2026Computer Environments Elicit General Agentic Intelligence in LLMs
  6. 2024Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
  7. 2025Guided Code Generation with LLMs: A Multi-Agent Framework for Complex Code Tasks
  8. 2024Large Language Model-Based Agents for Software Engineering: A Survey