mlsanityData to 7 Oct 2026

An Embodied Generalist Agent in 3D World

arXiv 2311.12871 ↗First posted 18 Nov 2023cs.CV cs.AI cs.CLcited by 355

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e.g., 3D grounding, embodied reasoning and acting. We argue these limitations significantly hinder current models from performing real-world tasks and approaching general intelligence. To this end, we introduce LEO, an embodied multi-modal generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world. LEO is trained with a unified task interface, model architecture, and objective in two stages: (i) 3D vision-language (VL) alignment and (ii) 3D vision-language-action (VLA) instruction tuning. We collect large-scale datasets comprising diverse object-level and scene-level tasks, which require considerable understanding of and interaction with the 3D world. Moreover, we meticulously design an LLM-assisted pipeline to produce high-quality 3D VL data. Through extensive experiments, we demonstrate LEO's remarkable proficiency across a wide spectrum of tasks, including 3D captioning, question answering, embodied reasoning, navigation and manipulation. Our ablative studies and scaling analyses further provide valuable insights for developing future embodied generalist agents. Code and data are available on project page.

Key facts. Cited by 355 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 83 of its 100 references that are in the map. First posted on arXiv 18 Nov 2023.

Builds on

83 of 100 references.

References · 83

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2023Segment Anythingcited by 8,158
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  13. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  14. 2015VQA: Visual Question Answeringcited by 2,489
  15. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  16. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  17. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  18. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  19. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  20. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  21. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  22. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  23. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  24. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  25. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  26. 2022A Generalist Agentcited by 802
  27. 2023Instruction Tuning with GPT-4cited by 676
  28. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  29. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  30. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  31. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  32. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  33. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  34. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  35. 2023LERF: Language Embedded Radiance Fieldscited by 538
  36. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  37. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  38. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 70; 2025: 173; 2026: 110.

355 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  4. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  5. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  6. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  7. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  8. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  9. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  10. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  11. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  12. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
38 more
  1. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  2. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  4. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  5. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  6. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  7. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  8. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  9. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  10. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  12. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  13. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  14. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  15. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  16. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  17. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  18. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  19. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  20. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  21. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
  22. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  23. 2025TesserAct: Learning 4D Embodied World Modelscited by 90
  24. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  25. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  26. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  27. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  28. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  29. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  30. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  31. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  32. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
  33. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  34. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  35. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 68
  36. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  37. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  38. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67

50 most cited of 355 All 355 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena
  2. 2026GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture
  3. 2026MAG-3D: Multi-Agent Grounded Reasoning for 3D Understanding
  4. 2025SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding
  5. 2025Benchmarking the Generality of Vision-Language-Action Models
  6. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
  7. 2026Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
  8. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models