# WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

arXiv: https://arxiv.org/abs/2608.04964 · HTML: https://mlsanity.com/paper/2608.04964/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 37)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024)
- [Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving](https://mlsanity.com/paper/2311.17918/) (2023)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023)
- [HPSv3: Towards Wide-Spectrum Human Preference Score](https://mlsanity.com/paper/2508.03789/) (2025)
- [WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling](https://mlsanity.com/paper/2512.14614/) (2025)
- [iVideoGPT: Interactive VideoGPTs are Scalable World Models](https://mlsanity.com/paper/2405.15223/) (2024)
- [Video World Models with Long-term Spatial Memory](https://mlsanity.com/paper/2506.05284/) (2025)
- [Epona: Autoregressive Diffusion World Model for Autonomous Driving](https://mlsanity.com/paper/2506.24113/) (2025)
- [IRASim: A Fine-Grained World Model for Robot Manipulation](https://mlsanity.com/paper/2406.14540/) (2024)
- [Pandora: Towards General World Model with Natural Language Actions and Video States](https://mlsanity.com/paper/2406.09455/) (2024)
- [Long-Context State-Space Video World Models](https://mlsanity.com/paper/2505.20171/) (2025)
- [HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds](https://mlsanity.com/paper/2604.14268/) (2026)
- [RLVR-World: Training World Models with Reinforcement Learning](https://mlsanity.com/paper/2505.13934/) (2025)
- [Policy Consolidation for Continual Reinforcement Learning](https://mlsanity.com/paper/1902.00255/) (2019)

## Built on by (most cited first, 3 of 3)

- [RECAP-Forcing: Retaining Content Appearances for Long Video Generation](https://mlsanity.com/paper/2608.26671/) (2026)
- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)
- [LoGo: Local-Global Rewards for Consistent Long-Horizon Video Generation](https://mlsanity.com/paper/2610.03636/) (2026)

## Abstract (arXiv)

Interactive video world models are essential for long-horizon planning and exploration, yet they suffer from compounding errors. Post-training methods such as reinforcement learning (RL) can improve these models, but they hit a verification bottleneck: for arbitrary action sequences, no ground-truth future state exists to measure long-term drift. Our key insight is that reversible action cycles make this verification possible: a sequence composed with its inverse must analytically return to the initial state, yielding annotation-free supervision on long-horizon correctness. Building on this, we introduce WorldCycle, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spatial closure reward enforcing symmetry between mirrored forward and reverse segments, and a temporal consistency reward aligning states across repeated cycle executions. These rewards force the model to learn actions as consistent state operators rather than memorized temporal patterns, and extend naturally to out-of-distribution composite action cycles that the base model handles poorly. We further release CycleBench, a diagnostic benchmark for state-returning ability under complex action structures. WorldCycle reduces state returning drift by up to 44% and lifts composite-action accuracy nearly 4x over the base model, providing a vital foundation for physically grounded world models.