# Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time arXiv: https://arxiv.org/abs/2606.15631 · HTML: https://mlsanity.com/paper/2606.15631/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. - Builds on 29 of its 30 references that are in the map. - First posted on arXiv 14 Jun 2026. ## Placed in - Vision-language-action models / [Reinforcement learning and post-training](https://mlsanity.com/vla/reinforcement-learning-and-post-training/) ## Builds on (most cited first, 29 of 29) - [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024) - [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023) - [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025) - [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023) - [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025) - [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025) - [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024) - [RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2506.18088/) (2025) - [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026) - [Fast-WAM: Do World Action Models Need Test-time Future Imagination?](https://mlsanity.com/paper/2603.16666/) (2026) - [Causal World Modeling for Robot Control](https://mlsanity.com/paper/2601.21998/) (2026) - [X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model](https://mlsanity.com/paper/2510.10274/) (2025) - [Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning](https://mlsanity.com/paper/2601.16163/) (2026) - [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023) - [Unified Video Action Model](https://mlsanity.com/paper/2503.00200/) (2025) - [MolmoAct: Action Reasoning Models that can Reason in Space](https://mlsanity.com/paper/2508.07917/) (2025) - [mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs](https://mlsanity.com/paper/2512.15692/) (2025) - [Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets](https://mlsanity.com/paper/2304.08742/) (2023) - [Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers](https://mlsanity.com/paper/2403.12943/) (2024) - [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026) - [STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learning](https://mlsanity.com/paper/2412.15182/) (2024) - [R+X: Retrieval and Execution from Everyday Human Videos](https://mlsanity.com/paper/2407.12957/) (2024) - [FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learning](https://mlsanity.com/paper/2408.16944/) (2024) - [EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Data](https://mlsanity.com/paper/2509.19626/) (2025) - [Residual Policy Learning for Shared Autonomy](https://mlsanity.com/paper/2004.05097/) (2020) - [MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos](https://mlsanity.com/paper/2509.09769/) (2025) - [ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection](https://mlsanity.com/paper/2405.03666/) (2024) - [HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval](https://mlsanity.com/paper/2505.20455/) (2025) - [Efficient and Reliable Teleoperation through Real-to-Sim-to-Real Shared Autonomy](https://mlsanity.com/paper/2603.17016/) (2026) ## Built on by (most cited first, 5 of 5) - [Retrieve in Time, Correct in Frequency](https://mlsanity.com/paper/2608.04527/) (2026) - [TraceFlow: Guiding Frozen Flow-Matching Robot Policies with Success and Failure Traces](https://mlsanity.com/paper/2609.20646/) (2026) - [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026) - [Reuse Before You Retrieve: Diagnosing Headroom and Complementarity for Test-Time Augmentation of Embodied Multimodal Policies](https://mlsanity.com/paper/2608.17484/) (2026) - [Learning Transferable World-Action Models from Task-Paired Human Videos](https://mlsanity.com/paper/2610.14351/) (2026) ## Abstract (arXiv) Extending a vision-language-action (VLA) policy to a new task typically requires task-specific teleoperated demonstrations and per-task fine-tuning, making adaptation costly in both data collection and compute. In this paper, we show that this target-side per-task adaptation cost can be replaced by retrieval. Our retrieval-augmented policy is trained once on paired demonstrations from the target embodiment (query) and a cheaper embodiment (pool, e.g., human-hand video), then frozen. New tasks are added at deployment by appending pool-side demonstrations to a retrieval pool. The frozen policy conditions on retrieved trajectories at every control step, so new tasks are absorbed by indexing data rather than updating parameters. Fine-tuning is needed only to take on a new, unseen embodiment, not for each new task. We show that retrieval improves policies beyond a specific backbone, including standard VLA policies, but its effect is especially pronounced in Cosmos Policy, a video-generation-based world-action model (WAM). In this setting, retrieval supplies coarse task progression, while the WAM's future-image objective provides an additional visual consistency signal that strengthens the retrieval-conditioned actions. On PushT, we study how retrieval provides a reusable high-level motion prior for cross-embodiment generalization to unseen goal angles, while on RoboTwin 2.0 our method outperforms cross-embodiment baselines on unseen tasks, and we additionally demonstrate the method on a real robot.