mlsanityData to 7 Oct 2026

An Interactive Agent Foundation Model

arXiv 2402.05929 ↗First posted 8 Feb 2024cs.AI cs.LG cs.ROcited by 25

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide range of applications. We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and tasks. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction, enabling a versatile and adaptable AI framework. We demonstrate the performance of our framework across three separate domains -- Robotics, Gaming AI, and Healthcare. Our model demonstrates its ability to generate meaningful and contextually relevant outputs in each area. The strength of our approach lies in its generality, leveraging a variety of data sources such as robotics sequences, gameplay data, large-scale video datasets, and textual information for effective multimodal and multi-task learning. Our approach provides a promising avenue for developing generalist, action-taking, multimodal systems.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 56 of its 71 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

56 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Agent AI: Surveying the Horizons of Multimodal Interaction7 Jan
  2. 2023Creative Robot Tool Use with Large Language Models19 Oct
  3. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation6 Mar

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2022A Generalist Agentcited by 802
  9. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  10. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  11. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  12. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  13. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  14. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  15. 2022Visual Language Maps for Robot Navigationcited by 362
  16. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  17. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  18. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  19. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  20. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  21. 2022Masked Visual Pre-training for Motor Controlcited by 246
  22. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  23. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  24. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  25. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  26. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  27. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  28. 2023Video Language Planningcited by 160
  29. 2023MindAgent: Emergent Gaming Interactioncited by 87
  30. 2023Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skillcited by 75
  31. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  32. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  33. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  34. 2023ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Applicationcited by 48
  35. 2023Interactive Task Planning with Language Modelscited by 36
  36. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  37. 2023Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuningcited by 15
  38. 2023Multitask Multimodal Prompted Training for Interactive Embodied Task Completioncited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 14; 2026: 3.

25 papers.

Sort:

By area:

  1. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  2. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  3. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  4. 2025PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathologycited by 36
  5. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  6. 2025Self-Improving Embodied Foundation Modelscited by 28
  7. 2025Towards Agentic AI Networking in 6G: A Generative Foundation Model-as-Agent Approachcited by 19
  8. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Modelscited by 14
  9. 2024Reflect-RL: Two-Player Online RL Fine-Tuning for LMscited by 13
  10. 2024Position: Foundation Agents as the Paradigm Shift for Decision Makingcited by 9
  11. 2025RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentationcited by 9
  12. 2025VLAs are Confined yet Capable of Generalizing to Novel Instructionscited by 8
13 more
  1. 2025Compressor-VLA: Instruction-Guided Visual Token Compression for Efficient Robotic Manipulationcited by 7
  2. 2024Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularitycited by 4
  3. 2024A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Modelcited by 4
  4. 2025PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agentscited by 4
  5. 2025How can we assess human-agent interactions? Case studies in software agent designcited by 4
  6. 2025Fetch.ai: An Architecture for Modern Multi-Agent Systemscited by 4
  7. 2025Completion neq Collaboration: Scaling Collaborative Effort with Agentscited by 3
  8. 2025Hydra: An Agentic Reasoning Approach for Enhancing Adversarial Robustness and Mitigating Hallucinations in Vision-Language Modelscited by 2
  9. 2026MagicAgent: Towards Generalized Agent Planningcited by 2
  10. 2024Geometric Algebra Meets Large Language Models: Instruction-Based Transformations of Separate Meshes in 3D, Interactive and Controllable Scenescited by 1
  11. 2024RoboGolf: Mastering Real-World Minigolf with a Reflective Multi-Modality Vision-Language Modelcited by 0
  12. 2026Prune, Update and Trim: Robust Structured Pruning for Large Language Modelscited by 0
  13. 2026Honest Lying: Understanding Memory Confabulation in Reflexive Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Position Paper: Agent AI Towards a Holistic Intelligence
  2. 2025Scaling Agents via Continual Pre-training
  3. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
  4. 2020Imitating Interactive Intelligence
  5. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
  6. 2025Single-Agent Scaling Fails Multi-Agent Intelligence: Towards Foundation Models with Native Multi-Agent Intelligence
  7. 2025Transforming Monolithic Foundation Models into Embodied Multi-Agent Architectures for Human-Robot Collaboration
  8. 2026InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning