mlsanityData to 14 Oct 2026

LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning

arXiv 2406.11815 ↗First posted 17 Jun 2024cs.RO cs.CV cs.LGcited by 87

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

In recent years, instruction-tuned Large Multimodal Models (LMMs) have been successful at several tasks, including image captioning and visual question answering; yet leveraging these models remains an open question for robotics. Prior LMMs for robotics applications have been extensively trained on language and action data, but their ability to generalize in different settings has often been less than desired. To address this, we introduce LLARVA, a model trained with a novel instruction tuning method that leverages structured prompts to unify a range of robotic learning tasks, scenarios, and environments. Additionally, we show that predicting intermediate 2-D representations, which we refer to as "visual traces", can help further align vision and action spaces for robot learning. We generate 8.5M image-visual trace pairs from the Open X-Embodiment dataset in order to pre-train our model, and we evaluate on 12 different tasks in the RLBench simulator as well as a physical Franka Emika Panda 7-DoF robot. Our experiments yield strong performance, demonstrating that LLARVA - using 2-D and language representations - performs well compared to several contemporary baselines, and can generalize across various robot environments and configurations.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 48 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

43 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2017Mask R-CNNcited by 7,277
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
30 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2015Fast R-CNNcited by 2,933
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  8. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  9. 2016Modeling Context in Referring Expressionscited by 1,063
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  12. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  13. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  14. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  15. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  16. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  17. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  18. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  19. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  20. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  21. 2019Connecting Vision and Language with Localized Narrativescited by 226
  22. 2018Videos as Space-Time Region Graphscited by 221
  23. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  24. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 122
  25. 2021Object-Region Video Transformerscited by 73
  26. 2018Object Level Visual Reasoning in Videoscited by 67
  27. 2019Learning Canonical Representations for Scene Graph to Image Generationcited by 60
  28. 2023Robot Learning with Sensorimotor Pre-trainingcited by 51
  29. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 41
  30. 2020Compositional Video Synthesis with Action Graphscited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 48; 2026: 29.

87 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  2. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 339
  3. 2024Latent Action Pretraining from Videoscited by 321
  4. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
  5. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  6. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  7. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  8. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
  9. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  10. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 105
  11. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  12. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 86
38 more
  1. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 81
  2. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 72
  3. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 70
  4. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  5. 2025Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgettingcited by 58
  6. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 56
  7. 2025Training Strategies for Efficient Embodied Reasoningcited by 55
  8. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 52
  9. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modificationcited by 49
  10. 2024Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoningcited by 47
  11. 2024A Dual Process VLA: Efficient Robotic Manipulation Leveraging VLMcited by 44
  12. 2025CEED-VLA: Consistency Vision-Language-Action Model with Early-Exit Decodingcited by 38
  13. 2025VLA-R1: Enhancing Reasoning in Vision-Language-Action Modelscited by 38
  14. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  15. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 35
  16. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 33
  17. 2025LLaDA-VLA: Vision Language Diffusion Action Modelscited by 30
  18. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  19. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 27
  20. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Modelscited by 27
  21. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMscited by 26
  22. 2025ViPRA: Video Prediction for Robot Actionscited by 26
  23. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representationscited by 25
  24. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  25. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstrationcited by 20
  26. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 19
  27. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulationcited by 18
  28. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  29. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  30. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  31. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  32. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  33. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulationcited by 14
  34. 2025NaviTrace: Evaluating Embodied Navigation of Vision-Language Modelscited by 13
  35. 2025PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policiescited by 12
  36. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  37. 2025Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulationcited by 11
  38. 2025RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Graspingcited by 11

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Improved Baselines with Visual Instruction Tuning
  2. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
  3. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Model
  5. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
  6. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  7. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Model
  8. 2026PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models