mlsanityData to 7 Oct 2026

Per-Group Error, Not Total MSE: Fine-Tuning Vision-Language-Action Models for 11-DoF Mobile Manipulation

arXiv 2606.00253 ↗First posted 29 May 2026cs.RO cs.LGcited by 0

On the map: Vision-language-action models → Generalist VLA models and benchmarks

Abstract (arXiv)

Fine-tuning Vision-Language-Action (VLA) models for mobile manipulators with heterogeneous joint spaces can produce a counterintuitive result: the checkpoint with the lowest aggregate MSE is not the one that performs best on the real robot. We argue this is a predictable consequence of collapsing heterogeneous joint groups (arm, gripper, head, wheeled base) into a single metric, where easy-to-predict joints can mask joints that still fail. We fine-tune SmolVLA (450M, action-expert only) on the 11-DoF Toyota HSR and compare it against π0.5 (3.3B), a stronger pretrained baseline. Per-group analysis exposes two patterns: in SmolVLA, the mobile base converges slowest and limits overall performance. In expert-only fine-tuning of π0.5 (training only the action head, backbone frozen), total MSE drops below the baseline but arm accuracy degrades. On 60 real-robot trials (20 per model), π0.5 80k (4.0/4) significantly outperforms both fine-tuned variants (expert-only 3k: 3.75/4; HSR-SmolVLA: 3.5/4; Mann-Whitney p ≤ 0.010), despite expert-only 3k having the lowest total MSE. This separation is most consistent with the offline arm-group error, not total MSE or base-group error. We conclude that per-group error is a more reliable signal than total MSE for checkpoint selection on robots with heterogeneous action spaces. Code: https://github.com/paumontagut/per-group-mse-vla

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models. Builds on 10 of its 12 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

10 of 12 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics2 Jun
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalization22 Apr
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Control31 Oct
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Model13 Jun
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 5

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  4. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  5. 2026LeRobot: An Open-Source Library for End-to-End Robot Learningcited by 70

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization
  2. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
  3. 2025Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation
  4. 2026MobileManiBench: Simplifying Model Verification for Mobile Manipulation
  5. 2026Reshaping Action Error Distributions for Reliable Vision-Language-Action Models
  6. 2026Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis
  7. 2026Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models
  8. 2025EdgeVLA: Efficient Vision-Language-Action Models