# TANDEM: Task and Motion Planning with As-Needed Demonstrations for Efficient Vision-Language-Action Model Fine-tuning

arXiv: https://arxiv.org/abs/2609.28314 · HTML: https://mlsanity.com/paper/2609.28314/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models.
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Placed in

- Vision-language-action models / [Reinforcement learning and post-training](https://mlsanity.com/vla/reinforcement-learning-and-post-training/)

## Builds on (most cited first, 27 of 27)

- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment](https://mlsanity.com/paper/2211.08416/) (2022)
- [DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning](https://mlsanity.com/paper/2502.16932/) (2025)
- [Imitating Task and Motion Planning with Visuomotor Transformers](https://mlsanity.com/paper/2305.16309/) (2023)
- [EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World](https://mlsanity.com/paper/2604.07607/) (2026)
- [SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment](https://mlsanity.com/paper/2410.18907/) (2024)
- [LazyDAgger: Reducing Context Switching in Interactive Imitation Learning](https://mlsanity.com/paper/2104.00053/) (2021)
- [Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware](https://mlsanity.com/paper/2505.09601/) (2025)
- [Masquerade: Learning from In-the-wild Human Videos using Data-Editing](https://mlsanity.com/paper/2508.09976/) (2025)
- [VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning](https://mlsanity.com/paper/2410.23156/) (2024)
- [RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction](https://mlsanity.com/paper/2509.07953/) (2025)
- [InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning](https://mlsanity.com/paper/2405.19758/) (2024)
- [Human-in-the-Loop Task and Motion Planning for Imitation Learning](https://mlsanity.com/paper/2310.16014/) (2023)
- [IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learning](https://mlsanity.com/paper/2405.01472/) (2024)
- [PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection](https://mlsanity.com/paper/2212.04708/) (2022)
- [MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation](https://mlsanity.com/paper/2603.16861/) (2026)
- [From Real World to Logic and Back: Learning Generalizable Relational Concepts For Long Horizon Robot Planning](https://mlsanity.com/paper/2402.11871/) (2024)
- [Open-World Task and Motion Planning via Vision-Language Model Generated Constraints](https://mlsanity.com/paper/2411.08253/) (2024)
- [TiPToP: A Modular Open-Vocabulary Robot Manipulation System That Plans](https://mlsanity.com/paper/2603.09971/) (2026)
- [Symskill: Symbol and Skill Co-Invention for Data-Efficient and Reactive Long-Horizon Manipulation](https://mlsanity.com/paper/2510.01661/) (2025)
- [SkillWrapper: Generative Predicate Invention for Task-level Robot Planning](https://mlsanity.com/paper/2511.18203/) (2025)
- [Set-Supervised Diffusion Policy: Learning Action-Chunking Diffusion through Corrections](https://mlsanity.com/paper/2606.01865/) (2026)
- [DATAFARM: Distribution-Aligned Task and Motion Planning for Fine-Tuning Vision-Language-Action Models](https://mlsanity.com/paper/2609.12316/) (2026)

## Abstract (arXiv)

Human teleoperators spend substantial time demonstrating behaviors that robots can already perform autonomously, limiting the scalability of data collection for robot foundation models. Task and motion planning (TAMP) can automate many of these behaviors, but a fixed planning domain may not support every stage of a long-horizon manipulation task. We present TANDEM (Tamp with As-Needed Demonstrations for Efficient Model fine-tuning), a system that combines TAMP with selective human teleoperation to collect demonstrations for tasks beyond the planner's capabilities. Our key idea is to represent human assistance as an on-demand planning capability. Given a language instruction and visual observation, TANDEM uses pretrained vision-language models to extend the planning domain with missing predicates and human-executed magic operators. This allows the planner to interleave autonomous and human-executed stages without task-specific intervention points. After each human stage, TANDEM re-perceives the scene and checks whether the intended effects hold before resuming autonomous planning. To support fine-tuning vision-language-action (VLA) models, TANDEM also uses example pretraining trajectories to align planner-generated motions with the target model's pretraining distribution. We evaluate TANDEM on five long-horizon manipulation tasks beyond the TAMP domain's capabilities. On a representative long-horizon task, TANDEM collects 2.9x as many demonstrations as full-task teleoperation at the same human intervention time. Fine-tuning a pretrained π0.5-DROID model on 20 TANDEM demonstrations per task increases average task success from 0% to 60% across the five tasks.