mlsanityData to 7 Oct 2026

Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations

arXiv 2511.22697 ↗First posted 27 Nov 2025cs.RO cs.CL cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics. Yet, unlike VLMs in the vision-language domain, VLAs for robotics require finetuning to contend with varying physical factors like robot embodiment, environment characteristics, and spatial relationships of each task. Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics. Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task. In this work, we introduce Robotic Steering, a finetuning approach grounded in mechanistic interpretability that leverages few-shot demonstrations to identify and selectively finetune task-specific attention heads aligned with the physical, visual, and linguistic requirements of robotic tasks. Through comprehensive on-robot evaluations with a Franka Emika robot arm, we demonstrate that Robotic Steering outperforms LoRA while achieving superior robustness under task variation, reduced computational cost, and enhanced interpretability for adapting VLAs to diverse robotic tasks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 69 references that are in the map. First posted on arXiv 27 Nov 2025.

Builds on

55 of 69 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations14 Sep
  2. 2024Vision Language Models are In-Context Value Learners7 Nov
  3. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning17 Jun
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  5. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  10. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  12. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
38 more
  1. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  2. 2022In-context Learning and Induction Headscited by 930
  3. 2022Toy Models of Superpositioncited by 928
  4. 2023Steering Language Models With Activation Engineeringcited by 824
  5. 2022A Generalist Agentcited by 802
  6. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  7. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  8. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  9. 2023In-Context Learning Creates Task Vectorscited by 302
  10. 2023Function Vectors in Large Language Modelscited by 292
  11. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  12. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  13. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  14. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  15. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  16. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  17. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  18. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  19. 2017Interpreting Deep Visual Representations via Network Dissectioncited by 103
  20. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 56
  21. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  22. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  23. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 48
  24. 2025ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Modelscited by 43
  25. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  26. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 32
  27. 2025Mechanistic interpretability for steering vision-language-action modelscited by 30
  28. 2024On Mechanistic Knowledge Localization in Text-to-Image Generative Modelscited by 27
  29. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  30. 2024Finding Visual Task Vectorscited by 24
  31. 2021Bayesian Meta-Learning for Few-Shot Policy Adaptation Across Robotic Platformscited by 22
  32. 2024Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIPcited by 19
  33. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 19
  34. 2025Data Retrieval with Importance Weights for Few-Shot Imitation Learningcited by 10
  35. 2024Contrast Sets for Evaluating Language-Guided Robot Policiescited by 7
  36. 2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalizationcited by 6
  37. 2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
  38. 2025Few-Shot Vision-Language Action-Incremental Policy Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026Observing and Controlling Features in Vision-Language-Action Modelscited by 14
  2. 2026VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracingcited by 9
  3. 2026CORAL: Scalable Multi-Task Robot Learning via LoRA Expertscited by 8
  4. 2026Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steeringcited by 4
  5. 2026Contrastive Conceptor Activation Steering (COAST): Unlocking Vision-Language-Action Models through Hidden Statescited by 3
  6. 2026Semi-Supervised Vision-Language-Action Modelcited by 3
  7. 2026Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Controlcited by 3
  8. 2026Embodied Interpretability: Linking Causal Understanding to Generalization in Vision-Language-Action Modelscited by 2
  9. 2026FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptationcited by 2
  10. 2026Revisiting Parameter Redundancy in Vision-Language-Action Models: Insights from VLM-to-VLA Adaptationcited by 2
  11. 2026SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibrationcited by 2
  12. 2026Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York Citycited by 0
3 more
  1. 2026Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Modelscited by 0
  2. 2026Natural State-Prediction Accuracy can Hide Weak Controlled Responsiveness in VLA Readoutscited by 0
  3. 2026MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
  2. 2026DiMaS: Distribution Matching for Steering Vision-Language-Action Models
  3. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review
  4. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
  5. 2026Stable Language Guidance for Vision-Language-Action Models
  6. 2025Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance
  7. 2026Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization
  8. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges