mlsanityData to 7 Oct 2026

Agent AI: Surveying the Horizons of Multimodal Interaction

arXiv 2401.03568 ↗First posted 7 Jan 2024cs.AI cs.HC cs.LGcited by 167

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Multi-modal AI systems will likely become a ubiquitous presence in our everyday lives. A promising approach to making these systems more interactive is to embody them as agents within physical and virtual environments. At present, systems leverage existing foundation models as the basic building blocks for the creation of embodied agents. Embedding agents within such environments facilitates the ability of models to process and interpret visual and contextual data, which is critical for the creation of more sophisticated and context-aware AI systems. For example, a system that can perceive user actions, human behavior, environmental objects, audio expressions, and the collective sentiment of a scene can be used to inform and direct agent responses within the given environment. To accelerate research on agent-based multimodal intelligence, we define "Agent AI" as a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data, and can produce meaningful embodied actions. In particular, we explore systems that aim to improve agents based on next-embodied action prediction by incorporating external knowledge, multi-sensory inputs, and human feedback. We argue that by developing agentic AI systems in grounded environments, one can also mitigate the hallucinations of large foundation models and their tendency to generate environmentally incorrect outputs. The emerging field of Agent AI subsumes the broader embodied and agentic aspects of multimodal interactions. Beyond agents acting and interacting in the physical world, we envision a future where people can easily create any virtual reality or simulated scene and interact with agents embodied within the virtual environment.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 110 of its 249 references that are in the map. First posted on arXiv 7 Jan 2024.

Builds on

110 of 249 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Creative Robot Tool Use with Large Language Models19 Oct
  2. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation20 Jun
  3. 2023CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation17 Jun
  4. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation6 Mar

Other references · 106

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2022Human Motion Diffusion Modelcited by 1,062
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  11. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  12. 2022A Generalist Agentcited by 802
  13. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  14. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  15. 2022A Survey on In-context Learningcited by 717
  16. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  17. 2018Object Hallucination in Image Captioningcited by 661
  18. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  19. 2018On Evaluation of Embodied Navigation Agentscited by 526
  20. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  21. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  22. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  23. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  24. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  25. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  26. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  27. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  28. 2023The False Promise of Imitating Proprietary LLMscited by 254
  29. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  30. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  31. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  32. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  33. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  34. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  35. 2022Learning Video Representations from Large Language Modelscited by 204
  36. 2023Large Language Models as General Pattern Machinescited by 196
  37. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 194
  38. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 90; 2026: 45.

167 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2024Large Language Models: A Surveycited by 506
  3. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  4. 2025Memory in the Age of AI Agentscited by 285
  5. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  7. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  10. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  11. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
38 more
  1. 2024Large Multimodal Agents: A Surveycited by 99
  2. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
  3. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
  4. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  5. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agentscited by 61
  6. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  7. 2025LLM Agents for Education: Advances and Applicationscited by 59
  8. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
  9. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  10. 2024Benchmarking Agentic Workflow Generationcited by 36
  11. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  12. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  13. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  14. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  15. 2024An Interactive Agent Foundation Modelcited by 25
  16. 2025Towards Agentic Recommender Systems in the Era of Multimodal Large Language Modelscited by 25
  17. 2025LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarioscited by 25
  18. 2024MouSi: Poly-Visual-Expert Vision-Language Modelscited by 24
  19. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  20. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  21. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  22. 2024StateAct: Enhancing LLM Base Agents via Self-prompting and State-trackingcited by 21
  23. 2025Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attackscited by 21
  24. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  25. 2024BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysiscited by 20
  26. 2025SkillFlow: Scalable and Efficient Agent Skill Retrieval Systemcited by 20
  27. 2025AutoMind: Adaptive Knowledgeable Agent for Automated Data Sciencecited by 20
  28. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 19
  29. 2025Towards Agentic AI Networking in 6G: A Generative Foundation Model-as-Agent Approachcited by 19
  30. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  31. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17
  32. 2024On the Intrinsic Self-Correction Capability of LLMs: Uncertainty and Latent Conceptcited by 16
  33. 2024Enhancing Investment Analysis: Optimizing AI-Agent Collaboration in Financial Researchcited by 14
  34. 2025The AI Agent Indexcited by 14
  35. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14
  36. 2025A Vision for Access Control in LLM-based Agent Systemscited by 14
  37. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  38. 2025Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigationcited by 13

50 most cited of 167 All 167 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Embodied AI Agents: Modeling the World
  2. 2026A Multimodal Framework for Human-Multi-Agent Interaction
  3. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning
  4. 2025Magma: A Foundation Model for Multimodal AI Agents
  5. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Review
  6. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
  7. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
  8. 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agents