# ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation

arXiv: https://arxiv.org/abs/2609.24563 · HTML: https://mlsanity.com/paper/2609.24563/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 26 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 25 of 25)

- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI](https://mlsanity.com/paper/2410.00425/) (2024)
- [Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2410.21845/) (2024)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins](https://mlsanity.com/paper/2504.13059/) (2025)
- [SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent](https://mlsanity.com/paper/2509.20414/) (2025)
- [SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes](https://mlsanity.com/paper/2602.09153/) (2026)
- [Re3Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation](https://mlsanity.com/paper/2502.08645/) (2025)
- [RaC: Robot Learning for Long-Horizon Tasks by Scaling Recovery and Correction](https://mlsanity.com/paper/2509.07953/) (2025)
- [ENPIRE: Agentic Robot Policy Self-Improvement in the Real World](https://mlsanity.com/paper/2606.19980/) (2026)
- [RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks](https://mlsanity.com/paper/2603.11558/) (2026)
- [SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation](https://mlsanity.com/paper/2606.28276/) (2026)
- [RoboSnap: One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation](https://mlsanity.com/paper/2607.06699/) (2026)
- [GraspGen-X: Cross-Embodiment 6-DOF Diffusion-based Grasping](https://mlsanity.com/paper/2606.00998/) (2026)
- [NeSyPack: A Neuro-Symbolic Framework for Bimanual Logistics Packing](https://mlsanity.com/paper/2506.06567/) (2025)
- [ManiSkillFormer: Demonstration-Free Compositional Manipulation via Geometric Contracts and Agentic Skill Graph](https://mlsanity.com/paper/2609.16331/) (2026)

## Built on by (most cited first, 1 of 1)

- [Splat2Act: From Captured Scenes to Personalized VLA Manipulation](https://mlsanity.com/paper/2610.13945/) (2026)

## Abstract (arXiv)

Adapting visuomotor policies to new manipulation tasks often requires substantial manual engineering or teleoperated data collection. Simulation can provide task-specific data at scale, but constructing the scene, designing expert behavior, and configuring data generation still require significant per-task effort. We present ARSTAG, an agentic Real2Sim2Real system that turns a single RGB image and a natural-language instruction directly into robot policy-learning data. A hierarchy of language agents constructs a task-scoped simulation scene, generates robot-feasible demonstrations, and expands the training distribution through task-consistent randomization, while a coordinator agent manages cross-stage feedback and recovery. Across seven manipulation tasks spanning grasping, placement, and stacking, the ARSTAG-generated demonstrations enable sim-to-real transfer of three visuomotor policy architectures to a dual-arm robot, with pi0.5 achieving an average real-world success rate of 74.6%. Ablations show that task-consistent randomization substantially improves robustness, and policy performance increases with generated dataset size. Project webpage: https://boweili666.github.io/ARSTAG/.