mlsanityData to 7 Oct 2026

Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning

arXiv 2408.14037 ↗First posted 26 Aug 2024cs.RO cs.LGcited by 44

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Increasingly large imitation learning datasets are being collected with the goal of training foundation models for robotics. However, despite the fact that data selection has been of utmost importance in vision and natural language processing, little work in robotics has questioned what data such models should actually be trained on. In this work we investigate how to weigh different subsets or ``domains'' of robotics datasets for robot foundation model pre-training. Concrete, we use distributionally robust optimization (DRO) to maximize worst-case performance across all possible downstream domains. Our method, Re-Mix, addresses the wide range of challenges that arise when applying DRO to robotics datasets including variability in action spaces and dynamics across different datasets. Re-Mix employs early stopping, action normalization, and discretization to counteract these issues. Through extensive experimentation on the largest open-source robot manipulation dataset, the Open X-Embodiment dataset, we demonstrate that data curation can have an outsized impact on downstream performance. Specifically, domain weights learned by Re-Mix outperform uniform weights by 38% on average and outperform human-selected weights by 32% on datasets used to train existing generalist robot policies, specifically the RT-X models.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 64 of its 77 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

64 of 77 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation4 Jan
  2. 2023BridgeData V2: A Dataset for Robot Learning at Scale24 Aug
  3. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learning18 Jul
  4. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation7 Jul
  5. 2023HYDRA: Hybrid Robot Actions for Imitation Learning29 Jun
  6. 2023Data Quality in Imitation Learning4 Jun
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  8. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets18 Apr
  9. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning20 Oct
  10. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors20 Oct
  11. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learning14 Oct
  12. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learning19 Sep
4 more
  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  2. 2021The Surprising Effectiveness of Representation Learning for Visual Imitation2 Dec
  3. 2020Visual Imitation Made Easy11 Aug
  4. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation7 Nov

Other references · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
36 more
  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  4. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  5. 2018Robustness May Be at Odds with Accuracycited by 873
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  10. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  11. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  12. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  13. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  14. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  15. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  16. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  17. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  18. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  19. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  20. 2023Demystifying CLIP Datacited by 265
  21. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  22. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  23. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  24. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  25. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  26. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  27. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  28. 2019Distributionally Robust Language Modelingcited by 142
  29. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  30. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  31. 2024Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approachcited by 48
  32. 2023Train Offline, Test Online: A Real Robot Learning Benchmarkcited by 47
  33. 2019Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Needcited by 43
  34. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 38
  35. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35
  36. 2023Polybot: Training One Policy Across Robots While Embracing Variabilitycited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 27; 2026: 14.

44 papers.

Sort:

By area:

  1. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  2. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  3. 2024Vision Language Models are In-Context Value Learnerscited by 84
  4. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  5. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 56
  6. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  7. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
  8. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
  9. 2025DataMIL: Selecting Data for Robot Imitation Learning with Datamodelscited by 25
  10. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 21
  11. 2025Model Predictive Task Sampling for Efficient and Robust Adaptationcited by 20
  12. 2025What Matters in Learning from Large-Scale Datasets for Robot Manipulationcited by 20
32 more
  1. 2025Is Diversity All You Need for Scalable Robotic Manipulation?cited by 19
  2. 2025Curating Demonstrations using Online Experiencecited by 16
  3. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 15
  4. 2025A Taxonomy for Evaluating Generalist Robot Manipulation Policiescited by 15
  5. 2025Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixelscited by 12
  6. 2025How to Train Your Robots? The Impact of Demonstration Modality on Imitation Learningcited by 11
  7. 2025Guiding Data Collection via Factored Scaling Curvescited by 10
  8. 2025Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoningcited by 10
  9. 2025Data Retrieval with Importance Weights for Few-Shot Imitation Learningcited by 10
  10. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 10
  11. 2025SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learningcited by 9
  12. 2025VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigationcited by 6
  13. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrationscited by 6
  14. 2025R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulationcited by 4
  15. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  16. 2026Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Roboticscited by 4
  17. 2026Learning from the Best: Smoothness-Driven Metrics for Data Quality in Imitation Learningcited by 3
  18. 2025A Hybrid Co-Finetuning Approach for Visual Bug Detection in Video Gamescited by 2
  19. 2025Hierarchical Vision Language Action Model Using Success and Failure Demonstrationscited by 2
  20. 2026Task Robustness via Re-Labelling Vision-Action Robot Datacited by 2
  21. 2026VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignmentcited by 2
  22. 2026SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulationcited by 2
  23. 2025FoldNet: Learning Generalizable Closed-Loop Policy for Garment Folding via Keypoint-Driven Asset and Demonstration Synthesiscited by 1
  24. 2026RIO: Flexible Real-Time Robot I/O for Cross-Embodiment Robot Learningcited by 1
  25. 2026FACTR 2: Learning External Force Sensing for Commodity Robot Arms Improves Policy Learningcited by 1
  26. 2026ATHENA: Accelerated Multi-Task Heterogeneous Influence Functions for Robot Data Curationcited by 1
  27. 2026Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steeringcited by 1
  28. 2025Using Temperature Sampling to Effectively Train Robot Learning Policies on Imbalanced Datasetscited by 0
  29. 2026SIEVE: Structure-Aware Data Selection for Imitation Learning with VLA Modelscited by 0
  30. 2026Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrationscited by 0
  31. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0
  32. 2026Imperfection for Precision: Upcycling Imperfect Data for High-Precision Robotic Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulation
  2. 2025Robot Data Curation with Mutual Information Estimators
  3. 2026Diversity You Can Actually Measure: A Fast, Model-Free Diversity Metric for Robotics Datasets
  4. 2026Data-Asymmetric Latent Imagination and Reranking for 3D Robotic Imitation Learning
  5. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitching
  6. 2025Generalization Capability for Imitation Learning
  7. 2024VITAL: Interactive Few-Shot Imitation Learning via Visual Human-in-the-Loop Corrections
  8. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations