mlsanityData to 13 Oct 2026

PaLM-E: An Embodied Multimodal Language Model

arXiv 2303.03378 ↗First posted 6 Mar 2023cs.LG cs.AI cs.ROcited by 2,052

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.

Key facts. Cited by 2,052 papers in the map (data through 13 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 71 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

63 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov
  2. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,321
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,220
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,028
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,452
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,369
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,294
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,186
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,116
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,065
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,032
  11. 2020Object-Centric Learning with Slot Attentioncited by 869
  12. 2022A Generalist Agentcited by 803
  13. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 764
  14. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 731
  15. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 693
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 659
  17. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  18. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 602
  19. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 546
  20. 2022Improving alignment of dialogue agents via targeted human judgementscited by 529
  21. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 518
  22. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  23. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  24. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 437
  25. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 404
  26. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 397
  27. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  28. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  29. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  30. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 224
  31. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  32. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  33. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  34. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  35. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  36. 2022Formal Mathematics Statement Curriculum Learningcited by 141
  37. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  38. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 408; 2024: 584; 2025: 524; 2026: 536.

2,052 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,827
  2. 2023Qwen Technical Reportcited by 3,310
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,022
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,823
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  7. 2023A Survey of Large Language Modelscited by 2,559
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,546
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,344
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,135
  11. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,699
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,637
38 more
  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,450
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,433
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,230
  5. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,197
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,092
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,082
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  9. 2023A Survey on Multimodal Large Language Modelscited by 905
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 898
  11. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 871
  12. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 769
  13. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 739
  14. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 708
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 706
  16. 2023VILA: On Pre-training for Visual Language Modelscited by 690
  17. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 687
  18. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 680
  19. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 678
  20. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 677
  21. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 656
  22. 2023The Internal State of an LLM Knows When It's Lyingcited by 652
  23. 2023A Comprehensive Overview of Large Language Modelscited by 644
  24. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  25. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 588
  26. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 586
  27. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 577
  28. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 563
  29. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 560
  30. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 551
  31. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  32. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 546
  33. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 540
  34. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 534
  35. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 523
  36. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 515
  37. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 503
  38. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501

50 most cited of 2,052 All 2,052 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Igniting VLMs toward the Embodied Space
  2. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
  3. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigation
  4. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
  5. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation
  6. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis
  7. 2026Thinker: A vision-language foundation model for embodied intelligence
  8. 2025EO-1: An Open Unified Embodied Foundation Model for General Robot Control