mlsanityData to 14 Oct 2026

Lemur: Harmonizing Natural Language and Code for Language Agents

arXiv 2310.06830 ↗First posted 10 Oct 2023cs.CLcited by 72

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We introduce Lemur and Lemur-Chat, openly accessible language models optimized for both natural language and coding capabilities to serve as the backbone of versatile language agents. The evolution from language chat models to functional language agents demands that models not only master human interaction, reasoning, and planning but also ensure grounding in the relevant environments. This calls for a harmonious blend of language and coding capabilities in the models. Lemur and Lemur-Chat are proposed to address this necessity, demonstrating balanced proficiencies in both domains, unlike existing open-source models that tend to specialize in either. Through meticulous pre-training using a code-intensive corpus and instruction fine-tuning on text and code data, our models achieve state-of-the-art averaged performance across diverse text and coding benchmarks among open-source models. Comprehensive experiments demonstrate Lemur's superiority over existing open-source models and its proficiency across various agent tasks involving human communication, tool usage, and interaction under fully- and partially- observable environments. The harmonization between natural and programming languages enables Lemur-Chat to significantly narrow the gap with proprietary models on agent abilities, providing key insights into developing advanced open-source agents adept at reasoning, planning, and operating seamlessly across environments. https://github.com/OpenLemur/Lemur

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 76 of its 99 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

76 of 99 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  2. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks27 May

Other references · 74

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  13. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  14. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  15. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  16. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  17. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  18. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  19. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  20. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  21. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  22. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  23. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  24. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  25. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  26. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  27. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  28. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  29. 2023Teaching Large Language Models to Self-Debugcited by 988
  30. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  31. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  32. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  33. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  34. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  35. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  36. 2022PAL: Program-aided Language Modelscited by 652
  37. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  38. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 34; 2025: 20; 2026: 7.

72 papers.

Sort:

By area:

  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  2. 2024AFlow: Automating Agentic Workflow Generationcited by 432
  3. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  4. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  5. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  6. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  7. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  8. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 177
  9. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  10. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  11. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 137
  12. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
38 more
  1. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  2. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  3. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  4. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  5. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 72
  6. 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 70
  7. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  8. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  9. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  10. 2024Large Language Model for Table Processing: A Surveycited by 52
  11. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 52
  12. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
  13. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
  14. 2025Contextual Experience Replay for Self-Improvement of Language Agentscited by 48
  15. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  16. 2024On the Multi-turn Instruction Following for Conversational Web Agentscited by 43
  17. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  18. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 42
  19. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  20. 2023LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystemcited by 38
  21. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solvingcited by 38
  22. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  23. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  24. 2024Empowering Large Language Model Agents through Action Learningcited by 22
  25. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 22
  26. 2024Sailor: Open Language Models for South-East Asiacited by 19
  27. 2026Internal Representations as Indicators of Hallucinations in Agent Tool Selectioncited by 19
  28. 2023INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repaircited by 18
  29. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  30. 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agentcited by 18
  31. 2024Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agentscited by 17
  32. 2023ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?cited by 15
  33. 2023SAGE: Smart home Agent with Grounded Executioncited by 13
  34. 2024Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMscited by 13
  35. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  36. 2024Collaborative Evolving Strategy for Automatic Data-Centric Developmentcited by 6
  37. 2025LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solvingcited by 6
  38. 2025Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Trainingcited by 6

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
  2. 2023Evaluating Language-Model Agents on Realistic Autonomous Tasks
  3. 2026Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite
  4. 2025Small Language Models are the Future of Agentic AI
  5. 2026GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactions
  6. 2025Why do AI agents communicate in human language?
  7. 2024Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication
  8. 2023ChatLogo: A Large Language Model-Driven Hybrid Natural-Programming Language Interface for Agent-based Modeling and Programming