# Robot Policy Evaluation for Sim-to-Real Transfer: A Benchmarking Perspective

arXiv: https://arxiv.org/abs/2508.11117 · HTML: https://mlsanity.com/paper/2508.11117/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2025.

## Built on by (most cited first, 5 of 5)

- [Interactive World Simulator for Robot Policy Training and Evaluation](https://mlsanity.com/paper/2603.08546/) (2026)
- [Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions](https://mlsanity.com/paper/2511.04665/) (2025)
- [NEBULA: Do We Evaluate Vision-Language-Action Agents Correctly?](https://mlsanity.com/paper/2510.16263/) (2025)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [When Does Test-Time Physical Diagnosis Pay? A Frozen Policy Buys Evidence It Never Reads](https://mlsanity.com/paper/2609.22299/) (2026)

## Abstract (arXiv)

Current vision-based robotics simulation benchmarks have significantly advanced robotic manipulation research. However, robotics is fundamentally a real-world problem, and evaluation for real-world applications has lagged behind in evaluating generalist policies. In this paper, we discuss challenges and desiderata in designing benchmarks for generalist robotic manipulation policies for the goal of sim-to-real policy transfer. We propose 1) utilizing high visual-fidelity simulation for improved sim-to-real transfer, 2) evaluating policies by systematically increasing task complexity and scenario perturbation to assess robustness, and 3) quantifying performance alignment between real-world performance and its simulation counterparts.