Robotic Assistant: Completing Collaborative Tasks with Dexterous Vision-Language-Action Models
On the map: Vision-language-action models → Human-video and latent-action pretraining
Abstract (arXiv)
We adapt a pre-trained Vision-Language-Action (VLA) model (Open-VLA) for dexterous human-robot collaboration with minimal language prompting. Our approach adds (i) FiLM conditioning to visual backbones for task-aware perception, (ii) an auxiliary intent head that predicts collaborator hand pose and target cues, and (iii) action-space post-processing that predicts compact deltas (position/rotation) and PCA-reduced finger joints before mapping to full commands. Using a multi-view, teleoperated Franka and Mimic-hand dataset augmented with MediaPipe hand poses, we demonstrate that delta actions are well-behaved and that four principal components explain ~96% of hand-joint variance. Ablations identify action post-processing as the primary performance driver; auxiliary intent helps, FiLM is mixed, and a directional motion loss is detrimental. A real-time stack (~0.3 s latency on one RTX 4090) composes "pick-up" and "pass" into a long-horizon behavior. We surface "trainer overfitting" to specific demonstrators as the key limitation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human-video and latent-action pretraining, one of 13 directions in Vision-language-action models. Builds on 22 of its 33 references that are in the map. First posted on arXiv 29 Oct 2025.
Builds on
22 of 33 references.
References · 22
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023Visual Instruction Tuningcited by 7,792
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
- 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
10 more
- 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
- 2023PaLM 2 Technical Reportcited by 1,154
- 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
- 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
- 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
- 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
- 2024LaMI: Large Language Models for Multi-Modal Human-Robot Interactioncited by 21
- 2019Imitating by generating: deep generative models for imitation of interactive taskscited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026InDex: Empowering VLA Models with Intent-Conditioned Arm-Hand Coordination for Dexterous Manipulation
- 2026Cross-Hand Latent Representation for Vision-Language-Action Models
- 2026Towards High-DoF Dexterous Manipulation through VLA Post-Training ↗
- 2026DexHiL: A Human-in-the-Loop Framework for Vision-Language-Action Model Post-Training in Dexterous Manipulation
- 2026MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalization
- 2026Learning to Assist: Collaborative VLAs for Implicit Human-Robot Collaboration ↗
- 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos
- 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos