mlsanityData to 7 Oct 2026

LLaRA: Supercharging Robot Learning Data for Vision-Language Policy

arXiv 2406.20095 ↗First posted 28 Jun 2024cs.RO cs.AI cs.CLcited by 51

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Vision Language Models (VLMs) have recently been leveraged to generate robotic actions, forming Vision-Language-Action (VLA) models. However, directly adapting a pretrained VLM for robotic control remains challenging, particularly when constrained by a limited number of robot demonstrations. In this work, we introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations and enables an efficient transfer of a pretrained VLM into a powerful VLA, motivated by the success of visual instruction tuning in Computer Vision. First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets, aligning robotic actions with image pixel coordinates. Further, we enhance this dataset in a self-supervised manner by defining six auxiliary tasks, without requiring any additional action annotations. We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control. Through experiments across multiple simulated and real-world tasks, we demonstrate that LLaRA achieves state-of-the-art performance while preserving the generalization capabilities of large language models. The code, datasets, and pretrained models are available at https://github.com/LostXine/LLaRA.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 82 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

64 of 82 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning17 Jun
  2. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  3. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  5. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  7. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2017Mask R-CNNcited by 7,259
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  6. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  8. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  9. 2022A Generalist Agentcited by 802
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  12. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  13. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  14. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  15. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  16. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  17. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  18. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  19. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  20. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  21. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  22. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  23. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  24. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  25. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  26. 2022Online Decision Transformercited by 194
  27. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  28. 2022Faithful Reasoning Using Large Language Modelscited by 130
  29. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  30. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  31. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  32. 2021Self-supervised Video Transformercited by 64
  33. 2021Predicting Video with VQVAEcited by 61
  34. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  35. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  36. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  37. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  38. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 28; 2026: 9.

51 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  3. 2024Latent Action Pretraining from Videoscited by 302
  4. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  5. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  6. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  7. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  8. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  9. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  10. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
  11. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  12. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
38 more
  1. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  2. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 68
  3. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 56
  4. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  5. 2024Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoningcited by 45
  6. 2024A Dual Process VLA: Efficient Robotic Manipulation Leveraging VLMcited by 44
  7. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 43
  8. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  9. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  10. 2025Pre-training Auto-regressive Robotic Models with 4D Representationscited by 27
  11. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  12. 2025ViPRA: Video Prediction for Robot Actionscited by 24
  13. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  14. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 17
  15. 2024Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Modelscited by 16
  16. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 16
  17. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  18. 2025MoTVLA: A Vision-Language-Action Model with Unified Fast-Slow Reasoningcited by 9
  19. 2024LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulationcited by 8
  20. 2025AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learningcited by 8
  21. 2026ST4VLA: Spatially Guided Training for Vision-Language-Action Modelscited by 8
  22. 2025Pixel Motion Diffusion is What We Need for Robot Controlcited by 7
  23. 2025Robotic VLA Benefits from Joint Learning with Motion Image Diffusioncited by 7
  24. 2025Focusing on What Matters: Object-Agent-centric Tokenization for Vision Language Action modelscited by 6
  25. 2025ROSA: Harnessing Robot States for Vision-Language and Action Alignmentcited by 5
  26. 2025Teaching Metric Distance to Discrete Autoregressive Language Modelscited by 3
  27. 2025Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigationcited by 3
  28. 2026UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Modelscited by 3
  29. 2026Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Taskscited by 3
  30. 2025Towards Fast, Memory-based and Data-Efficient Vision-Language Policycited by 2
  31. 2025VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Modelscited by 2
  32. 2025Gondola: Grounded Vision Language Planning for Robotic Manipulationcited by 1
  33. 2026IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidancecited by 1
  34. 2024CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representationcited by 0
  35. 2025BLAZER: Bootstrapping LLM-based Manipulation Agents with Zero-Shot Data Generationcited by 0
  36. 2026Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoningcited by 0
  37. 2026Gaze-to-text Generation: Beyond Categorical Decoding of Human Attentioncited by 0
  38. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaDA-VLA: Vision Language Diffusion Action Models
  2. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model
  3. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulation
  4. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation
  5. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  6. 2025Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting
  7. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations
  8. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data