# Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trends

arXiv: https://arxiv.org/abs/2606.12207 · HTML: https://mlsanity.com/paper/2606.12207/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 111 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents](https://mlsanity.com/paper/2502.09560/) (2025)
- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots](https://mlsanity.com/paper/2306.14874/) (2023)
- [SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents](https://mlsanity.com/paper/2412.13178/) (2024)
- [RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots](https://mlsanity.com/paper/2603.04356/) (2026)
- [GRUtopia: Dream General Robots in a City at Scale](https://mlsanity.com/paper/2407.10943/) (2024)
- [EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents](https://mlsanity.com/paper/2501.11858/) (2025)
- [A Survey on Robotics with Foundation Models: toward Embodied AI](https://mlsanity.com/paper/2402.02385/) (2024)
- [GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs](https://mlsanity.com/paper/2410.03645/) (2024)
- [RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation](https://mlsanity.com/paper/2507.00435/) (2025)
- [RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies](https://mlsanity.com/paper/2510.17950/) (2025)
- [EFM3D: A Benchmark for Measuring Progress Towards 3D Egocentric Foundation Models](https://mlsanity.com/paper/2406.10224/) (2024)
- [AeroVerse: UAV-Agent Benchmark Suite for Simulating, Pre-training, Finetuning, and Evaluating Aerospace Embodied World Models](https://mlsanity.com/paper/2408.15511/) (2024)
- [InfiniteWorld: A Unified Scalable Simulation Framework for General Visual-Language Robot Interaction](https://mlsanity.com/paper/2412.05789/) (2024)
- [BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks](https://mlsanity.com/paper/2410.12974/) (2024)
- [Audit Cards: Contextualizing AI Evaluations](https://mlsanity.com/paper/2504.13839/) (2025)
- [Toward Embodied AGI: A Review of Embodied AI and the Road Ahead](https://mlsanity.com/paper/2505.14235/) (2025)
- [AnyBody: A Benchmark Suite for Cross-Embodiment Manipulation](https://mlsanity.com/paper/2505.14986/) (2025)
- [GPAI Evaluations Standards Taskforce: Towards Effective AI Governance](https://mlsanity.com/paper/2411.13808/) (2024)
- [EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems](https://mlsanity.com/paper/2604.11174/) (2026)
- [AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation](https://mlsanity.com/paper/2604.11674/) (2026)

## Built on by (most cited first, 1 of 1)

- [Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence](https://mlsanity.com/paper/2609.29669/) (2026)

## Abstract (arXiv)

Embodied intelligence now spans navigation, household assistance, manipulation, autonomous driving, aerial agents, and multimodal large-model control. This expansion has made benchmark construction a central bottleneck for reliable evaluation. Unlike static datasets, embodied benchmarks combine task specifications, environments, robot data, demonstrations, annotations, metrics, evaluation scripts, and release policies into a single evaluation system. This survey reviews the literature through a five-stage construction pipeline: requirement and task construction, data acquisition, data cleaning and annotation, benchmark suite generation and metric definition, and evaluation execution with diagnostic feedback. For each stage, the survey analyzes the transition from manual curation to traditional automation, foundation-model assistance, and agentic closed-loop workflows. It also compares qualitative construction costs across human labor, data and asset acquisition, compute and simulation, validation and debugging, governance and maintenance, and rework risk. The main conclusion is that automation does not simply reduce benchmark cost. Instead, it often shifts cost toward validation, auditability, version control, and long-term governance. Progress in embodied evaluation will therefore depend not only on larger benchmark suites, but also on construction pipelines that are diagnosable, auditable, and responsibly refreshable.