mlsanityData to 7 Oct 2026

Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robust

arXiv 2410.01971 ↗First posted 2 Oct 2024cs.RO cs.LGcited by 33

On the map: Vision-language-action models → Generalist VLA models and benchmarks

Abstract (arXiv)

Vision-language-action (VLA) models trained on large-scale internet data and robot demonstrations have the potential to serve as generalist robot policies. However, despite their large-scale training, VLAs are often brittle to task-irrelevant visual details such as distractor objects or background colors. We introduce Bring Your Own VLA (BYOVLA): a run-time intervention scheme that (1) dynamically identifies regions of the input image that the model is sensitive to, and (2) minimally alters task-irrelevant regions to reduce the model's sensitivity using automated image editing tools. Our approach is compatible with any off the shelf VLA without model fine-tuning or access to the model's weights. Hardware experiments on language-instructed manipulation tasks demonstrate that BYOVLA enables state-of-the-art VLA models to nearly retain their nominal performance in the presence of distractor objects and backgrounds, which otherwise degrade task success rates by up to 40%. Website with additional information, videos, and code: https://aasherh.github.io/byovla/ .

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models. Builds on 44 of its 50 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

44 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Model13 Jun
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Segment Anythingcited by 8,158
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  11. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
30 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  4. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  5. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  8. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  9. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  10. 2024Scaling and evaluating sparse autoencoderscited by 530
  11. 2020Reinforcement Learning with Augmented Datacited by 452
  12. 2017Interpretation of Neural Networks is Fragilecited by 443
  13. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  14. 2017The (Un)reliability of saliency methodscited by 331
  15. 2023Inpaint Anything: Segment Anything Meets Image Inpaintingcited by 233
  16. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  17. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  18. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  19. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  20. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  21. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  22. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 60
  23. 2020Neuroevolution of Self-Interpretable Agentscited by 48
  24. 2024RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learningcited by 46
  25. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  26. 2023Transferring Foundation Models for Generalizable Robotic Manipulationcited by 18
  27. 2020Learning Task-Driven Control Policies via Information Bottleneckscited by 13
  28. 2023MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learningcited by 9
  29. 2023ROSO: Improving Robotic Policy Inference via Synthetic Observationscited by 3
  30. 2023Less is more -- the Dispatcher/ Executor principle for multi-task Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 21.

33 papers.

Sort:

By area:

  1. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  2. 2025Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgettingcited by 52
  3. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  4. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  5. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 33
  6. 2025RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbationscited by 23
  7. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 16
  8. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 13
  9. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
  10. 2025PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policiescited by 11
  11. 2025Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Groundingcited by 10
  12. 2026VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracingcited by 9
21 more
  1. 2026STRONG-VLA: Decoupled Robustness Learning for Vision-Language-Action Models under Multimodal Perturbationscited by 8
  2. 2025Reimagination with Test-time Observation Interventions: Distractor-Robust World Model Predictions for Visual Model Predictive Controlcited by 6
  3. 2026DTP: A Simple yet Effective Distracting Token Pruning Framework for Vision-Language Action Modelscited by 5
  4. 2026ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Controlcited by 5
  5. 2025IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex taskscited by 4
  6. 2026Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steeringcited by 4
  7. 2025Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondencecited by 3
  8. 2026ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulationcited by 3
  9. 2026CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Modelscited by 3
  10. 2026Overcoming Visual Clutter in Vision Language Action Models via Concept-Gated Visual Distillationcited by 2
  11. 2026ActFovea: Runtime Safeguarding for VLA Policies via Spatiotemporal Visual-Action Consistencycited by 2
  12. 2026What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Modelscited by 0
  13. 2026Green for Go, Red for No: Visual Grounding via Semantic Segmentation for VLA Navigation Policiescited by 0
  14. 2026Security of Foundation-Model-Powered Embodied Agents: Attack Surfaces, Attacks, Defenses, and Evaluationcited by 0
  15. 2026DeicticVLA: Unifying Instruction Modes Based on Language and Deictic Gestures in a Single VLAcited by 0
  16. 2026Sensing Which Modality Matters: Evidence-Gated Regularization for Robust VLA Policiescited by 0
  17. 2026What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policiescited by 0
  18. 2026Are Vision-Language-Action Models Robust to One-Step Observation Perturbations?cited by 0
  19. 2026ActionUNet: Improving Robustness of VLA Models with Efficient Multi-scale Fine-tuningcited by 0
  20. 2026Taming VLAs under Robot Execution Errors: Self-Compensation and Stress Testingcited by 0
  21. 2026RawVLA: Embodied Neural Image Signal Processor For Robotic Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation
  2. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model
  3. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks
  4. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review
  5. 2025EdgeVLA: Efficient Vision-Language-Action Models
  6. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modification
  7. 2026VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models
  8. 2025Unveiling the Potential of Vision-Language-Action Models with Open-Ended Multimodal Instructions