# Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models

arXiv: https://arxiv.org/abs/2605.01896 · HTML: https://mlsanity.com/paper/2605.01896/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 41 of its 65 references that are in the map.
- First posted on arXiv 3 May 2026.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)

## Builds on (most cited first, 30 of 41)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [Matrix-game 2.0: An open-source real-time and streaming interactive world model](https://mlsanity.com/paper/2508.13009/) (2025)
- [GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2503.20523/) (2025)
- [History-Guided Video Diffusion](https://mlsanity.com/paper/2502.06764/) (2025)
- [WorldMem: Long-term Consistent World Simulation with Memory](https://mlsanity.com/paper/2504.12369/) (2025)
- [GameFactory: Creating New Games with Generative Interactive Videos](https://mlsanity.com/paper/2501.08325/) (2025)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models](https://mlsanity.com/paper/2505.23656/) (2025)
- [Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling](https://mlsanity.com/paper/2507.07982/) (2025)
- [TesserAct: Learning 4D Embodied World Models](https://mlsanity.com/paper/2504.20995/) (2025)
- [Video as the New Language for Real-World Decision Making](https://mlsanity.com/paper/2402.17139/) (2024)
- [SpatialVID: A Large-Scale Video Dataset with Spatial Annotations](https://mlsanity.com/paper/2509.09676/) (2025)
- [OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling](https://mlsanity.com/paper/2509.12201/) (2025)

## Abstract (arXiv)

Emerging multi-modal world models attempt to jointly generate videos across diverse modalities (e.g., RGB, depth, and mask), yet they fail to fully exploit the rich priors of existing foundation models. We propose M2-REPA, the first representation alignment method tailored for multi-modal video generation. Our key insight is that foundation models trained on different modality spaces naturally capture distinct domain-specific priors, acting as complementary "experts." Specifically, we first decouple modality-specific features from the diffusion model's intermediate representations, then align each with its corresponding expert foundation model. To this end, we design two synergistic objectives: a multi-modal representation alignment loss that enforces feature-to-expert matching, and a modality-specific decoupling regularization that encourages complementarity across different modalities. This design enables joint optimization, fully exploiting priors from multiple foundation models. Extensive experiments demonstrate that our method significantly outperforms baselines in visual quality and long-term consistency.