mlsanityData to 7 Oct 2026

Towards A Unified Agent with Foundation Models

arXiv 2307.09668 ↗First posted 18 Jul 2023cs.RO cs.AI cs.LGcited by 46

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Language Models and Vision Language Models have recently demonstrated unprecedented capabilities in terms of understanding human intentions, reasoning, scene understanding, and planning-like behaviour, in text form, among many others. In this work, we investigate how to embed and leverage such abilities in Reinforcement Learning (RL) agents. We design a framework that uses language as the core reasoning tool, exploring how this enables an agent to tackle a series of fundamental RL challenges, such as efficient exploration, reusing experience data, scheduling skills, and learning from observations, which traditionally require separate, vertically designed algorithms. We test our method on a sparse-reward simulated robotic manipulation environment, where a robot needs to stack a set of objects. We demonstrate substantial performance improvements over baselines in exploration efficiency and ability to reuse data from offline datasets, and illustrate how to reuse learned skills to solve novel tasks or imitate videos of human experts.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

46 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents29 Jan
  2. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2017Deep reinforcement learning from human preferencescited by 3,846
32 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  11. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  12. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  13. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  14. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  15. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  16. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  17. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  18. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  19. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  20. 2017Teacher-Student Curriculum Learningcited by 197
  21. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  22. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  23. 2018Self-Imitation Learningcited by 135
  24. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  25. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  26. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  27. 2023Collaborating with language models for embodied reasoningcited by 61
  28. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  29. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26
  30. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
  31. 2021Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Explorationcited by 14
  32. 2021Manipulator-Independent Representations for Visual Imitationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 24; 2025: 4; 2026: 6.

46 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023Cognitive Architectures for Language Agentscited by 422
  3. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  4. 2023Language is All a Graph Needscited by 219
  5. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  6. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  7. 2024Embodied Understanding of Driving Scenarioscited by 63
  8. 2023Language Models as Zero-Shot Trajectory Generatorscited by 59
  9. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  10. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  11. 2024Benchmarking Agentic Workflow Generationcited by 36
  12. 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32
34 more
  1. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  2. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  3. 2025LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Modelscited by 21
  4. 2023Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMscited by 19
  5. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planningcited by 19
  6. 2023Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Owncited by 16
  7. 2023KwaiAgents: Generalized Information-seeking Agent System with Large Language Modelscited by 14
  8. 2024Automating the Enterprise with Foundation Modelscited by 14
  9. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  10. 2024Bootstrapping Cognitive Agents with a Large Language Modelcited by 9
  11. 2024NEUSIS: A Compositional Neuro-Symbolic Framework for Autonomous Perception, Reasoning, and Planning in Complex UAV Search Missionscited by 8
  12. 2024Multi-granularity Knowledge Transfer for Continual Reinforcement Learningcited by 6
  13. 2024VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Trainingcited by 5
  14. 2024MaPPing Your Model: Assessing the Impact of Adversarial Attacks on LLM-based Programming Assistantscited by 5
  15. 2024MaxMI: A Maximal Mutual Information Criterion for Manipulation Concept Discoverycited by 5
  16. 2024Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learningcited by 5
  17. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  18. 2024Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Modelscited by 4
  19. 2024BBSEA: An Exploration of Brain-Body Synchronization for Embodied Agentscited by 3
  20. 2024InfoCon: Concept Discovery with Generative and Discriminative Informativenesscited by 3
  21. 2026Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Surveycited by 3
  22. 2026LookasideVLN: Direction-Aware Aerial Vision-and-Language Navigationcited by 3
  23. 2024Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Modelscited by 2
  24. 2024View From Above: A Framework for Evaluating Distribution Shifts in Model Behaviorcited by 2
  25. 2024Interacting Large Language Model Agents. Interpretable Models and Social Learningcited by 2
  26. 2023Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perceptioncited by 1
  27. 2024Game On: Towards Language Models as RL Experimenterscited by 1
  28. 2024The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requestscited by 1
  29. 2025Integrating Machine Learning into Belief-Desire-Intention Agents: Current Advances and Open Challengescited by 1
  30. 2026Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunitiescited by 1
  31. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  32. 2026Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settingscited by 0
  33. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learningcited by 0
  34. 2026What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Words as Beacons: Guiding RL Agents with High-Level Language Prompts
  2. 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
  3. 2025GoalLadder: Incremental Goal Discovery with Vision-Language Models
  4. 2025PARL: Prompt-based Agents for Reinforcement Learning
  5. 2025Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation with Large Language Models
  6. 2026Exploration and Exploitation Errors Are Measurable for Language Model Agents
  7. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
  8. 2024From Goal-Conditioned to Language-Conditioned Agents via Vision-Language Models