# Assistron: Bayesian Shared Autonomy with Off-the-shelf Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2606.23147 · HTML: https://mlsanity.com/paper/2606.23147/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models.
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Placed in

- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [Real-Time Execution of Action Chunking Flow Policies](https://mlsanity.com/paper/2506.07339/) (2025)
- [MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2508.19236/) (2025)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning](https://mlsanity.com/paper/2109.08273/) (2021)
- [Training-free Linear Image Inverses via Flows](https://mlsanity.com/paper/2310.04432/) (2023)
- [DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning](https://mlsanity.com/paper/2502.16932/) (2025)
- [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](https://mlsanity.com/paper/2506.18123/) (2025)
- [On the Guidance of Flow Matching](https://mlsanity.com/paper/2502.02150/) (2025)
- [Running VLAs at Real-time Speed](https://mlsanity.com/paper/2510.26742/) (2025)
- [Shared Autonomy via Hindsight Optimization for Teleoperation and Teaming](https://mlsanity.com/paper/1706.00155/) (2017)
- [Inference-Time Policy Steering through Human Interactions](https://mlsanity.com/paper/2411.16627/) (2024)
- [To the Noise and Back: Diffusion for Shared Autonomy](https://mlsanity.com/paper/2302.12244/) (2023)
- [PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection](https://mlsanity.com/paper/2212.04708/) (2022)
- [Learning a Thousand Tasks in a Day](https://mlsanity.com/paper/2511.10110/) (2025)
- [Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition](https://mlsanity.com/paper/2407.00299/) (2024)
- [End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collection](https://mlsanity.com/paper/2511.00139/) (2025)
- [Robot Trajectron: Trajectory Prediction-based Shared Control for Robot Manipulation](https://mlsanity.com/paper/2402.02499/) (2024)
- [AR-VLA: True Autoregressive Action Expert for Vision-Language-Action Models](https://mlsanity.com/paper/2603.10126/) (2026)
- [Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA](https://mlsanity.com/paper/2603.08122/) (2026)
- [On Optimizing Interventions in Shared Autonomy](https://mlsanity.com/paper/2112.09169/) (2021)
- [Shared Autonomy with IDA: Interventional Diffusion Assistance](https://mlsanity.com/paper/2409.15317/) (2024)
- [DiSCo: Diffusion Sequence Copilots for Shared Autonomy](https://mlsanity.com/paper/2603.22787/) (2026)
- [Assistive Planning in Complex, Dynamic Environments: a Probabilistic Approach](https://mlsanity.com/paper/1506.06784/) (2015)
- [Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models](https://mlsanity.com/paper/2506.14727/) (2025)

## Abstract (arXiv)

We propose Assistron, a shared autonomy model that leverages Vision-Language-Action (VLA) models to assist the user in daily activities. Our approach is grounded in two core principles: (1)~minimizing human cognitive and physical effort by leveraging VLA-driven autonomy for macro-movements, and (2)~prioritizing human intervention specifically at critical failure points. Driven by the user's verbal language commands, Assistron utilizes the VLA to autonomously execute macro-reaching trajectories, saving users' effort. In contact-rich interactions where VLAs tend to fail, Assistron employs a phase-aware interaction detection mechanism and solicits the user to intervene, in turn adjusting the VLA's action generation via flow matching guidance. Critically, our formulation eliminates the need for VLA fine-tuning, protecting its broad behavioral priors from catastrophic forgetting and ensuring the model does not become a narrow specialist. We validate our approach on a comprehensive multi-task scene recovery benchmark encompassing diverse daily manipulation skills. Empirical results demonstrate that Assistron significantly improves task success rates over pure autonomous baselines while significantly reducing human cognitive and physical workload compared to traditional teleoperation, offering a scalable, smooth, and effortless paradigm for assistive manipulation. The code is available in https://github.com/mousecpn/Assistron.git.