mlsanityData to 7 Oct 2026

Large Language Models as Generalizable Policies for Embodied Tasks

arXiv 2310.17722 ↗First posted 26 Oct 2023cs.LG cs.AI cs.CLcited by 86

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We show that large language models (LLMs) can be adapted to be generalizable policies for embodied visual tasks. Our approach, called Large LAnguage model Reinforcement Learning Policy (LLaRP), adapts a pre-trained frozen LLM to take as input text instructions and visual egocentric observations and output actions directly in the environment. Using reinforcement learning, we train LLaRP to see and act solely through environmental interactions. We show that LLaRP is robust to complex paraphrasings of task instructions and can generalize to new tasks that require novel optimal behavior. In particular, on 1,000 unseen tasks it achieves 42% success rate, 1.7x the success rate of other common learned baselines or zero-shot applications of LLMs. Finally, to aid the community in studying language conditioned, massively multi-task, embodied AI problems we release a novel benchmark, Language Rearrangement, consisting of 150,000 training and 1,000 testing tasks for language-conditioned rearrangement. Video examples of LLaRP in unseen Language Rearrangement instructions are at https://llm-rl.github.io.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 75 of its 108 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

75 of 108 references.

References · 75

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
38 more
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  13. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  14. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  15. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  16. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  17. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  18. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  19. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  20. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  21. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  22. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  23. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  24. 2022Visual Language Maps for Robot Navigationcited by 362
  25. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  26. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  27. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  28. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  29. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  30. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  31. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  32. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  33. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  34. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  35. 2022Online Decision Transformercited by 194
  36. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  37. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  38. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 21; 2025: 46; 2026: 16.

86 papers.

Sort:

By area:

  1. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  2. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  3. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  4. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  5. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  6. 2024Vision Language Models are In-Context Value Learnerscited by 84
  7. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  8. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  9. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  10. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videoscited by 51
  11. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 49
  12. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
38 more
  1. 2024Grounding Multimodal Large Language Models in Actionscited by 35
  2. 2024Agents Thinking Fast and Slow: A Talker-Reasoner Architecturecited by 35
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  4. 2024GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robotcited by 28
  5. 2025Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspectivecited by 28
  6. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  7. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  8. 2025FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Tradingcited by 23
  9. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  10. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  11. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedbackcited by 19
  12. 2025NeSyC: A Neuro-symbolic Continual Learner For Complex Embodied Tasks In Open Domainscited by 15
  13. 2024Eurekaverse: Environment Curriculum Generation via Large Language Modelscited by 14
  14. 2024Reflect-RL: Two-Player Online RL Fine-Tuning for LMscited by 13
  15. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  16. 2025Enhancing Multi-Agent Systems via Reinforcement Learning with LLM-based Planner and Graph-based Policycited by 12
  17. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agentscited by 11
  18. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learningcited by 10
  19. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9
  20. 2024Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examplescited by 9
  21. 2025Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learningcited by 9
  22. 2025World-aware Planning Narratives Enhance Large Vision-Language Model Plannercited by 9
  23. 2026AdaWorldPolicy: World-Model-Driven Diffusion Policy with Online Adaptive Learning for Robotic Manipulationcited by 9
  24. 2025Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learningcited by 8
  25. 2025LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systemscited by 8
  26. 2025Structured Preference Optimization for Vision-Language Long-Horizon Task Planningcited by 7
  27. 2025Enhancing Decision-Making of Large Language Models via Actor-Criticcited by 7
  28. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 7
  29. 2024Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rolloutscited by 6
  30. 2024Benchmarking General-Purpose In-Context Learningcited by 6
  31. 2025DSADF: Thinking Fast and Slow for Decision Makingcited by 6
  32. 2025ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Taskscited by 6
  33. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 6
  34. 2025Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model Systemcited by 5
  35. 2025VLA Model-Expert Collaboration for Bi-directional Manipulation Learningcited by 5
  36. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5
  37. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  38. 2024LightHouse: A Survey of AGI Hallucinationcited by 4

50 most cited of 86 All 86 (JSON)