mlsanityData to 7 Oct 2026

Enhancing Linguistic Generalization of VLA: Fine-Tuning OpenVLA via Synthetic Instruction Augmentation

arXiv 2603.16044 ↗First posted 17 Mar 2026cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-evolving LLM agents

Abstract (arXiv)

Generalization remains a core challenge in embodied AI, as robots must adapt to diverse environments. While OpenVLA represents the State-of-the-Art (SOTA) in Vision-Language-Action models by leveraging large-scale pre-training, its zero-shot performance can be limited when encountering completely new environments. This paper proposes a parameter-efficient fine-tuning strategy to enhance the linguistic generalization of OpenVLA by synthesizing a general instruction set for the Bridge Dataset V2. The paper leverages a Large Language Model (LLM) to generate a rich variety of semantically equivalent but structurally diverse commands for existing trajectories. In this experiment, Low-Rank Adaptation (LoRA) is implemented to fine-tune OpenVLA on augmented pairs, allowing the model to bridge the gap between complex natural language intent and robotic actions. Results demonstrate that the LoRA-enhanced model's robustness, suggesting that enriching the linguistic space of specialized datasets is crucial for embodied agents.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-evolving LLM agents, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 37 of its 44 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

37 of 44 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
25 more
  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  2. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  3. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  4. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  5. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  6. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  7. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  8. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  9. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  10. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  11. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  12. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  13. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  14. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  15. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  16. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  17. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  18. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  19. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  20. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  21. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 124
  22. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  23. 2023Robot Learning with Sensorimotor Pre-trainingcited by 51
  24. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47
  25. 2023Polybot: Training One Policy Across Robots While Embracing Variabilitycited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  2. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
  3. 2024A Survey on Vision-Language-Action Models for Embodied AI
  4. 2025What Can RL Bring to VLA Generalization? An Empirical Study
  5. 2026Decoupling the Declarative from the Procedural in Vision-Language-Action Models
  6. 2025Towards Accessible Physical AI: LoRA-Based Fine-Tuning of VLA Models for Real-World Robot Control
  7. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modification
  8. 2026CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts