mlsanityData to 14 Oct 2026

Multi-Task Interactive Robot Fleet Learning with Visual World Models

arXiv 2410.22689 ↗First posted 30 Oct 2024cs.RO cs.AIcited by 36

On the map: World models → Embodied and robot world models

Abstract (arXiv)

Recent advancements in large-scale multi-task robot learning offer the potential for deploying robot fleets in household and industrial settings, enabling them to perform diverse tasks across various environments. However, AI-enabled robots often face challenges with generalization and robustness when exposed to real-world variability and uncertainty. We introduce Sirius-Fleet, a multi-task interactive robot fleet learning framework to address these challenges. Sirius-Fleet monitors robot performance during deployment and involves humans to correct the robot's actions when necessary. We employ a visual world model to predict the outcomes of future actions and build anomaly predictors to predict whether they will likely result in anomalies. As the robot autonomy improves, the anomaly predictors automatically adapt their prediction criteria, leading to fewer requests for human intervention and gradually reducing human workload over time. Evaluations on large-scale benchmarks demonstrate Sirius-Fleet's effectiveness in improving multi-task policy performance and monitoring accuracy. We demonstrate Sirius-Fleet's performance in both RoboCasa in simulation and Mutex in the real world, two diverse, large-scale multi-task benchmarks. More information is available on the project website: https://ut-austin-rpl.github.io/sirius-fleet

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 38 of its 50 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

38 of 50 references.

References · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  9. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  10. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  11. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  12. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
26 more
  1. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 266
  2. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  3. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 204
  4. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 175
  5. 2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
  6. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 112
  7. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 111
  8. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 108
  9. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 108
  10. 2018EnsembleDAgger: A Bayesian Approach to Safe Imitation Learningcited by 100
  11. 2023The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systemscited by 88
  12. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 87
  13. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  14. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  15. 2021Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulationcited by 59
  16. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 51
  17. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 49
  18. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 38
  19. 2022A System-Level View on Out-of-Distribution Data in Roboticscited by 33
  20. 2022Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervisioncited by 30
  21. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 25
  22. 2021Correct Me if I am Wrong: Interactive Learning for Robotic Manipulationcited by 24
  23. 2019Scaled Autonomy: Enabling Human Operators to Control Robot Fleetscited by 16
  24. 2023Asking for Help: Failure Prediction in Behavioral Cloning through Value Approximationcited by 16
  25. 2023Closing the Loop on Runtime Monitors with Fallback-Safe MPCcited by 14
  26. 2024Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failurescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 23.

36 papers.

Sort:

By area:

  1. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 80
  2. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 52
  3. 2025Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policiescited by 52
  4. 2025Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysiscited by 45
  5. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 44
  6. 2025Failure Prediction at Runtime for Generative Robot Policiescited by 42
  7. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  8. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 14
  9. 2026Foundational World Models Accurately Detect Bimanual Manipulator Failurescited by 11
  10. 2026Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latentscited by 11
  11. 2025MoE-DP: An MoE-Enhanced Diffusion Policy for Robust Long-Horizon Robotic Manipulation with Skill Decomposition and Failure Recoverycited by 10
  12. 2025SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learningcited by 9
24 more
  1. 2026Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoringcited by 9
  2. 2025Genie Centurion: Accelerating Scalable Real-World Robot Training with Human Rewind-and-Refine Guidancecited by 7
  3. 2026Failure Identification in Imitation Learning Via Statistical and Semantic Filteringcited by 6
  4. 2026Overcoming Statistical Bias in Action-Controllable World Modelscited by 6
  5. 2026When to Act, Ask, or Learn: Uncertainty-Aware Policy Steeringcited by 5
  6. 2026FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Spacecited by 4
  7. 2025Predictive Preference Learning from Human Interventionscited by 3
  8. 2026ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policiescited by 3
  9. 2026Never Too Late for Force: Accelerating VLA Post-Training with Reactive Force Injectioncited by 3
  10. 2026PATCH: Action-Chunk-Conditioned Latent Patch Innovation Monitoring for Robot Manipulationcited by 2
  11. 2026SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibrationcited by 2
  12. 2026ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Modelscited by 2
  13. 2026Runtime Monitoring of Perception-Based Autonomous Systems via Embedding Temporal Logiccited by 1
  14. 2026VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Modelscited by 1
  15. 2026GUARD: Grounding Uncertainty and Ablation-Based Risk Detection for Diffusion-Based VLAscited by 1
  16. 2025LangPert: Detecting and Handling Task-level Perturbations for Robust Object Rearrangementcited by 0
  17. 2025Uncertainty-Guided Expert-AI Collaboration for Efficient Soil Horizon Annotationcited by 0
  18. 2026Perturbation-Based Epistemic Uncertainty for Failure Detection in Vision-Language-Action Modelscited by 0
  19. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0
  20. 2026DistAL: Distance-based Advantage Learning for VLA Fine-Tuningcited by 0
  21. 2026SEES: A Self-Evolving Embodied System via Failure-Guided VLA Policy Adaptationcited by 0
  22. 2026Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Modelscited by 0
  23. 2026Action-Consequence Alignment for Reliable Planning and Self-Improving in Latent World Modelscited by 0
  24. 2026I3L: Iterative Interactive Imitation Learning for Whole-Body Mobile Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DeepFleet: Multi-Agent Foundation Models for Mobile Robots
  2. 2025RobotFleet: An Open-Source Framework for Centralized Multi-Robot Task Planning
  3. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies
  4. 2026Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models
  5. 2024RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-World
  6. 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots
  7. 2026SPACE: Enabling Learning from Cross-Robot Data Toward Generalist Policies
  8. 2024Bayesian Optimization Framework for Efficient Fleet Design in Autonomous Multi-Robot Exploration