# SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation

arXiv: https://arxiv.org/abs/2606.18610 · HTML: https://mlsanity.com/paper/2606.18610/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning and post-training, one of 13 directions in Vision-language-action models.
- Builds on 24 of its 25 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Placed in

- Vision-language-action models / [Reinforcement learning and post-training](https://mlsanity.com/vla/reinforcement-learning-and-post-training/)

## Builds on (most cited first, 24 of 24)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning](https://mlsanity.com/paper/2601.16163/) (2026)
- [Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets](https://mlsanity.com/paper/2504.02792/) (2025)
- [Unified Video Action Model](https://mlsanity.com/paper/2503.00200/) (2025)
- [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025)
- [WorldMem: Long-term Consistent World Simulation with Memory](https://mlsanity.com/paper/2504.12369/) (2025)
- [Ctrl-World: A Controllable Generative World Model for Robot Manipulation](https://mlsanity.com/paper/2510.10125/) (2025)
- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [IRASim: A Fine-Grained World Model for Robot Manipulation](https://mlsanity.com/paper/2406.14540/) (2024)
- [WorldGym: World Model as An Environment for Policy Evaluation](https://mlsanity.com/paper/2506.00613/) (2025)
- [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](https://mlsanity.com/paper/2506.18123/) (2025)
- [Evaluating Gemini Robotics Policies in a Veo World Simulator](https://mlsanity.com/paper/2512.10675/) (2025)
- [PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies](https://mlsanity.com/paper/2512.16881/) (2025)
- [Interactive World Simulator for Robot Policy Training and Evaluation](https://mlsanity.com/paper/2603.08546/) (2026)
- [Scalable Policy Evaluation with Video World Models](https://mlsanity.com/paper/2511.11520/) (2025)
- [RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies](https://mlsanity.com/paper/2604.09860/) (2026)
- [World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry](https://mlsanity.com/paper/2604.01985/) (2026)
- [World Models That Know When They Don't Know - Controllable Video Generation with Calibrated Uncertainty](https://mlsanity.com/paper/2512.05927/) (2025)
- [dWorldEval: Scalable Robotic Policy Evaluation via Discrete Diffusion World Model](https://mlsanity.com/paper/2604.22152/) (2026)

## Built on by (most cited first, 7 of 7)

- [Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence](https://mlsanity.com/paper/2607.07675/) (2026)
- [WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation](https://mlsanity.com/paper/2608.09298/) (2026)
- [IMPLY: Physically Anchored Consistency for World-Model Rollouts](https://mlsanity.com/paper/2609.12441/) (2026)
- [RoboCoach: World Models as Active Coaches for Compositional Robot Skills](https://mlsanity.com/paper/2609.39685/) (2026)
- [Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics](https://mlsanity.com/paper/2610.13522/) (2026)
- [Data-Free Continual Learning for World Action Model with Rectified Generative Replay](https://mlsanity.com/paper/2610.18038/) (2026)
- [WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions](https://mlsanity.com/paper/2610.18826/) (2026)

## Abstract (arXiv)

Evaluating generalist robot manipulation policies in the real world is expensive, slow, and difficult to scale. Action-conditioned video world models offer a scalable alternative by simulating policy rollouts. Autoregressive rollouts accumulate compounding errors, observations across multiple camera views must remain mutually consistent, and the evaluator must generalize to policies whose behaviors lie outside the training distribution. We address these challenges with SC3-Eval, a self-consistent video generation recipe that adapts a pre-trained video foundation model into an accurate policy evaluator by enforcing three complementary forms of consistency. First, forward-inverse dynamics consistency jointly trains the model to predict frames from actions and to recover actions from frames, anchoring generated rollouts to a physically plausible action manifold and counteracting the drift a forward-only model cannot penalize. Second, cross-view consistency trains the model to inpaint each camera view from the other, keeping the multi-camera observation coherent over long rollouts without any explicit memory mechanism. Third, test-time consistency reuses the inverse dynamics mode at inference as a per-action-chunk uncertainty signal that terminates rollouts whose generated frames drift away from the requested actions. We also demonstrate SC3-Eval rollouts reproduce the failure modes that policies exhibit in real-world rollouts, supporting fine-grained diagnostic comparison rather than aggregate ranking alone. Across seven real-world vision-language-action policies, SC3-Eval attains a closed-loop Pearson correlation of 0.929 and MMRV of 0.119, outperforming three strong prior video-model-based baselines, and generalizes to new tasks.