mlsanityData to 7 Oct 2026

TANDEM: Task and Motion Planning with As-Needed Demonstrations for Efficient Vision-Language-Action Model Fine-tuning

arXiv 2609.28314 ↗First posted 23 Sep 2026cs.ROcited by 0

On the map: Vision-language-action models → Reinforcement learning and post-training also fits: Reasoning and chain-of-thought in VLA

Abstract (arXiv)

Human teleoperators spend substantial time demonstrating behaviors that robots can already perform autonomously, limiting the scalability of data collection for robot foundation models. Task and motion planning (TAMP) can automate many of these behaviors, but a fixed planning domain may not support every stage of a long-horizon manipulation task. We present TANDEM (Tamp with As-Needed Demonstrations for Efficient Model fine-tuning), a system that combines TAMP with selective human teleoperation to collect demonstrations for tasks beyond the planner's capabilities. Our key idea is to represent human assistance as an on-demand planning capability. Given a language instruction and visual observation, TANDEM uses pretrained vision-language models to extend the planning domain with missing predicates and human-executed magic operators. This allows the planner to interleave autonomous and human-executed stages without task-specific intervention points. After each human stage, TANDEM re-perceives the scene and checks whether the intended effects hold before resuming autonomous planning. To support fine-tuning vision-language-action (VLA) models, TANDEM also uses example pretraining trajectories to align planner-generated motions with the target model's pretraining distribution. We evaluate TANDEM on five long-horizon manipulation tasks beyond the TAMP domain's capabilities. On a representative long-horizon task, TANDEM collects 2.9x as many demonstrations as full-task teleoperation at the same human intervention time. Fine-tuning a pretrained π0.5-DROID model on 20 TANDEM demonstrations per task increases average task success from 0% to 60% across the five tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models. Builds on 27 of its 31 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

27 of 31 references.

References · 27

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  3. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  4. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  5. 2020Integrated Task and Motion Planningcited by 281
  6. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  7. 2025DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learningcited by 64
  8. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  9. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50
  10. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  11. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  12. 2025Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardwarecited by 41
15 more
  1. 2025Masquerade: Learning from In-the-wild Human Videos using Data-Editingcited by 38
  2. 2024VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planningcited by 37
  3. 2025RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correctioncited by 35
  4. 2024InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planningcited by 31
  5. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learningcited by 28
  6. 2024IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learningcited by 24
  7. 2022PATO: Policy Assisted TeleOperation for Scalable Robot Data Collectioncited by 19
  8. 2026MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulationcited by 11
  9. 2024From Real World to Logic and Back: Learning Generalizable Relational Concepts For Long Horizon Robot Planningcited by 9
  10. 2024Open-World Task and Motion Planning via Vision-Language Model Generated Constraintscited by 9
  11. 2026TiPToP: A Modular Open-Vocabulary Robot Manipulation System That Planscited by 7
  12. 2025Symskill: Symbol and Skill Co-Invention for Data-Efficient and Reactive Long-Horizon Manipulationcited by 3
  13. 2025SkillWrapper: Generative Predicate Invention for Task-level Robot Planningcited by 2
  14. 2026Set-Supervised Diffusion Policy: Learning Action-Chunking Diffusion through Correctionscited by 1
  15. 2026DATAFARM: Distribution-Aligned Task and Motion Planning for Fine-Tuning Vision-Language-Action Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptation
  2. 2026Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models
  3. 2026HIL-UMI: Bringing Human-in-the-Loop Post-Training of Vision-Language-Action Models to Universal Manipulation Interface
  4. 2025Dual-Actor Fine-Tuning of VLA Models: A Talk-and-Tweak Human-in-the-Loop Approach
  5. 2025Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins
  6. 2026PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models
  7. 2026SAPS: Shared Autonomy for Policy Steering by Blending Teleoperation with a Pretrained VLA
  8. 2026PAPO-VLA: Planning-Aware Policy Optimization for Vision-Language-Action Models