# Prepare Before You Act: Learning From Humans to Rearrange Initial States

arXiv: https://arxiv.org/abs/2509.18043 · HTML: https://mlsanity.com/paper/2509.18043/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 28 of 28)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators](https://mlsanity.com/paper/2309.13037/) (2023)
- [Data Scaling Laws in Imitation Learning for Robotic Manipulation](https://mlsanity.com/paper/2410.18647/) (2024)
- [Flow as the Cross-Domain Manipulation Interface](https://mlsanity.com/paper/2407.15208/) (2024)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2405.01527/) (2024)
- [AdaWorld: Learning Adaptable World Models with Latent Actions](https://mlsanity.com/paper/2503.18938/) (2025)
- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [BAKU: An Efficient Transformer for Multi-Task Policy Learning](https://mlsanity.com/paper/2406.07539/) (2024)
- [CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning](https://mlsanity.com/paper/2212.05711/) (2022)
- [OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation](https://mlsanity.com/paper/2403.07870/) (2024)
- [Local Policies Enable Zero-shot Long-horizon Manipulation](https://mlsanity.com/paper/2410.22332/) (2024)
- [Semantically Controllable Augmentations for Generalizable Robot Learning](https://mlsanity.com/paper/2409.00951/) (2024)
- [RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrations](https://mlsanity.com/paper/2411.16959/) (2024)
- [Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy for Visuomotor Imitation Learning](https://mlsanity.com/paper/2411.03294/) (2024)
- [RECON: Reducing Causal Confusion with Human-Placed Markers](https://mlsanity.com/paper/2409.13607/) (2024)
- [CIVIL: Causal and Intuitive Visual Imitation Learning](https://mlsanity.com/paper/2504.17959/) (2025)
- [Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondence](https://mlsanity.com/paper/2506.12678/) (2025)
- [OmniD: Generalizable Robot Manipulation Policy via Image-Based BEV Representation](https://mlsanity.com/paper/2508.11898/) (2025)

## Built on by (most cited first, 3 of 3)

- [A Survey on Efficient Vision-Language-Action Models](https://mlsanity.com/paper/2510.24795/) (2025)
- [StageCraft: Execution Aware Mitigation of Distractor and Obstruction Failures in VLA Models](https://mlsanity.com/paper/2603.20659/) (2026)
- [RoboAware: Learning to Coordinate Embodied Skills from Counterfactual Outcomes](https://mlsanity.com/paper/2610.11480/) (2026)

## Abstract (arXiv)

Imitation learning (IL) has proven effective across a wide range of manipulation tasks. However, IL policies often struggle when faced with out-of-distribution observations; for instance, when the target object is in a previously unseen position or occluded by other objects. In these cases, extensive demonstrations are needed for current IL methods to reach robust and generalizable behaviors. But when humans are faced with these sorts of atypical initial states, we often rearrange the environment for more favorable task execution. For example, a person might rotate a coffee cup so that it is easier to grasp the handle, or push a box out of the way so they can directly grasp their target object. In this work we seek to equip robot learners with the same capability: enabling robots to prepare the environment before executing their given policy. We propose ReSET, an algorithm that takes initial states -- which are outside the policy's distribution -- and autonomously modifies object poses so that the restructured scene is similar to training data. Theoretically, we show that this two step process (rearranging the environment before rolling out the given policy) reduces the generalization gap. Practically, our ReSET algorithm combines action-agnostic human videos with task-agnostic teleoperation data to i) decide when to modify the scene, ii) predict what simplifying actions a human would take, and iii) map those predictions into robot action primitives. Comparisons with diffusion policies, VLAs, and other baselines show that using ReSET to prepare the environment enables more robust task execution with equal amounts of total training data. See videos at our project website: https://reset2025paper.github.io/