mlsanityData to 7 Oct 2026

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models

arXiv 2311.05997 ↗First posted 10 Nov 2023cs.AIcited by 170

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. JARVIS-1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., "chopping trees" to long-horizon tasks, e.g., "obtaining a diamond pickaxe". JARVIS-1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task of textttObtainDiamondPickaxe, JARVIS-1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. The project page is available at https://craftjarvis.org/JARVIS-1

Key facts. Cited by 170 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 48 of its 65 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

48 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory25 May
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
33 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2023Teaching Large Language Models to Self-Debugcited by 963
  7. 2022A Generalist Agentcited by 802
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  11. 2023Augmented Language Models: a Surveycited by 398
  12. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  13. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  14. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  15. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  16. 2023Continual Pre-training of Language Modelscited by 194
  17. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  18. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  19. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  20. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  21. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  22. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  23. 2023MindAgent: Emergent Gaming Interactioncited by 87
  24. 2023Collaborating with language models for embodied reasoningcited by 61
  25. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  26. 2021The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 40
  27. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  28. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  29. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Controlcited by 32
  30. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
  31. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26
  32. 2019The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 25
  33. 2021SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competitioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 80; 2025: 42; 2026: 43.

170 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  3. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  4. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  5. 2023Large Language Models for Robotics: A Surveycited by 132
  6. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 120
  7. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  8. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 108
  9. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  10. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
  11. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  12. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
38 more
  1. 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 100
  2. 2024Large Multimodal Agents: A Surveycited by 99
  3. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  4. 2024I-Design: Personalized LLM Interior Designercited by 87
  5. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  6. 2024AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planningcited by 81
  7. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  8. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 79
  9. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  10. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
  11. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  12. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  13. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 60
  14. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  15. 2025OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflowscited by 54
  16. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  17. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  18. 2024Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Modelscited by 45
  19. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  20. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  21. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  22. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  23. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  24. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  25. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 37
  26. 2024Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacycited by 36
  27. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  28. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  29. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  30. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  31. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  32. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  33. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  34. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  35. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 24
  36. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  37. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policycited by 24
  38. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 23

50 most cited of 170 All 170 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023See and Think: Embodied Agent in Virtual Environment
  2. 2026MineNPC-Task: Task Suite for Memory-Aware Minecraft Agents
  3. 2025Training Agents Inside of Scalable World Models
  4. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning
  5. 2025Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agents
  6. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedback
  7. 2026CEDAR: Automata as Verifiable Interfaces for Language-Guided Embodied Action
  8. 2024STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft