# Guiding Data Collection via Factored Scaling Curves

arXiv: https://arxiv.org/abs/2505.07728 · HTML: https://mlsanity.com/paper/2505.07728/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 12 May 2025.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023)
- [ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI](https://mlsanity.com/paper/2410.00425/) (2024)
- [RegMix: Data Mixture as Regression for Language Model Pre-training](https://mlsanity.com/paper/2407.01492/) (2024)
- [ALOHA Unleashed: A Simple Recipe for Robot Dexterity](https://mlsanity.com/paper/2410.13126/) (2024)
- [Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment](https://mlsanity.com/paper/2211.08416/) (2022)
- [Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation](https://mlsanity.com/paper/1810.07121/) (2018)
- [DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning](https://mlsanity.com/paper/2502.16932/) (2025)
- ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023)
- [Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets](https://mlsanity.com/paper/2304.08742/) (2023)
- [Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning](https://mlsanity.com/paper/2408.14037/) (2024)
- [Efficient Data Collection for Robotic Manipulation via Compositional Generalization](https://mlsanity.com/paper/2403.05110/) (2024)
- [Robot Data Curation with Mutual Information Estimators](https://mlsanity.com/paper/2502.08623/) (2025)
- [Eliciting Compatible Demonstrations for Multi-Human Imitation Learning](https://mlsanity.com/paper/2210.08073/) (2022)
- [Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks](https://mlsanity.com/paper/2411.05821/) (2024)
- [On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation](https://mlsanity.com/paper/2312.12345/) (2023)
- [Predictive Red Teaming: Breaking Policies Without Breaking Robots](https://mlsanity.com/paper/2502.06575/) (2025)

## Built on by (most cited first, 12 of 12)

- [EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World](https://mlsanity.com/paper/2604.07607/) (2026)
- [RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction](https://mlsanity.com/paper/2509.07953/) (2025)
- [What Matters in Learning from Large-Scale Datasets for Robot Manipulation](https://mlsanity.com/paper/2506.13536/) (2025)
- [LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer](https://mlsanity.com/paper/2602.10556/) (2026)
- [REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation](https://mlsanity.com/paper/2512.19562/) (2025)
- [R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation](https://mlsanity.com/paper/2510.08547/) (2025)
- [Grounding Robot Generalization in Training Data via Retrieval-Augmented VLMs](https://mlsanity.com/paper/2603.11426/) (2026)
- [Active Real-World Factor-Based Evaluation for Generalist Robot Policies](https://mlsanity.com/paper/2607.14439/) (2026)
- [Teach and Grow: An Agent-Centered Architecture for General Robot Learning](https://mlsanity.com/paper/2608.17209/) (2026)
- [Anatomy of a Closed-Loop Collapse: A Causal Case Study of a Compressed VLA Policy](https://mlsanity.com/paper/2609.23048/) (2026)
- [PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training](https://mlsanity.com/paper/2610.04616/) (2026)
- [Training Generalist Navigation Policies via Action Self-Distillation](https://mlsanity.com/paper/2610.17593/) (2026)

## Abstract (arXiv)

Generalist imitation learning policies trained on large datasets show great promise for solving diverse manipulation tasks. However, to ensure generalization to different conditions, policies need to be trained with data collected across a large set of environmental factor variations (e.g., camera pose, table height, distractors) - a prohibitively expensive undertaking, if done exhaustively. We introduce a principled method for deciding what data to collect and how much to collect for each factor by constructing factored scaling curves (FSC), which quantify how policy performance varies as data scales along individual or paired factors. These curves enable targeted data acquisition for the most influential factor combinations within a given budget. We evaluate the proposed method through extensive simulated and real-world experiments, across both training-from-scratch and fine-tuning settings, and show that it boosts success rates in real-world tasks in new environments by up to 26% over existing data-collection strategies. We further demonstrate how factored scaling curves can effectively guide data collection using an offline metric, without requiring real-world evaluation at scale.