Encoder Winners Do Not Reliably Transfer Across VLA Backbone Scale: A Frozen-Backbone Grafting Diagnostic
Not on the map. Why?
Abstract (arXiv)
Vision-language-action (VLA) policies typically inherit their vision encoder from upstream VLM releases, but it is unclear whether an encoder choice validated on a small VLA transfers to a larger backbone. We introduce a frozen-backbone grafting diagnostic: the vision tower of a released VLA is replaced by a candidate encoder under a fixed protocol (adaptive average pooling, LayerNorm, and a single trainable linear projector), with the language model and action expert frozen. Across four encoders, two LIBERO suites, two backbones (SmolVLA-450M and π0.5-3.3B), and two-to-three seeds per cell (40 main grafting runs plus native, LoRA, pooling, and zero-/shuffled-image controls, all scored by offline action MSE), the small-backbone winner does not reliably select the large-backbone top tier: SigLIP is best on SmolVLA across both suites, while on π0.5 DINOv2-small leads the spatial suite and the object suite is a seed-sensitive near-tie band; three of the four backbone-suite comparisons (and 11 of 12 seed-level cells) support backbone-dependent rankings. The grafting wrapper is itself non-neutral with opposite sign across backbones (+45-56% MSE on the SmolVLA native tower, -50-52% on π0.5), so all conclusions are conditional on the fixed grafting protocol. We position frozen grafting as a cheap target-backbone diagnostic to run before committing to an encoder at scale, not as a closed-loop deployment claim.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 17 references that are in the map. First posted on arXiv 12 Jun 2026.
Builds on
14 of 17 references.
References · 14
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2016Layer Normalizationcited by 5,669
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
- 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
- 2024PaliGemma: A versatile 3B VLM for transfercited by 696
- 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 303
- 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
- 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 162
- 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
2 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026What Makes an Efficient VLA? Navigating Action-Head Design, Scaling, and Latency
- 2026Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders
- 2026The Layer Mystery of VLA: An Information-Theoretical Analysis of VLA Latent Interface
- 2025Zero-Shot Vision Encoder Grafting via LLM Surrogates
- 2026Not All Layers Need Tuning: Diagnosing and Directing Adaptation in Vision-Language-Action Models
- 2026Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Models
- 2026Retrieve in Time, Correct in Frequency
- 2026Decoupled Early Exits for Task-Dependent Compute Allocation in Flow-Matching VLAs