# MotionStrata: Hierarchical Motion Latents for Compact Video Autoencoding

arXiv: https://arxiv.org/abs/2506.07136 · HTML: https://mlsanity.com/paper/2506.07136/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Builds on (most cited first, 28 of 28)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Open-Sora Plan: Open-Source Large Video Generation Model](https://mlsanity.com/paper/2412.00131/) (2024)
- [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://mlsanity.com/paper/2204.03638/) (2022)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [iVideoGPT: Interactive VideoGPTs are Scalable World Models](https://mlsanity.com/paper/2405.15223/) (2024)
- [OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation](https://mlsanity.com/paper/2406.09399/) (2024)
- [Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization](https://mlsanity.com/paper/2402.03161/) (2024)
- [Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks](https://mlsanity.com/paper/1709.07592/) (2017)
- [SODA: Bottleneck Diffusion Models for Representation Learning](https://mlsanity.com/paper/2311.17901/) (2023)
- [CV-VAE: A Compatible Video VAE for Latent Generative Video Models](https://mlsanity.com/paper/2405.20279/) (2024)
- [WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.17459/) (2024)
- [ElasticTok: Adaptive Tokenization for Image and Video](https://mlsanity.com/paper/2410.08368/) (2024)
- [Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition](https://mlsanity.com/paper/2403.14148/) (2024)
- [Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization](https://mlsanity.com/paper/2503.11056/) (2025)
- [Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space](https://mlsanity.com/paper/2505.17011/) (2025)
- [REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latents](https://mlsanity.com/paper/2411.13552/) (2024)
- [UniAvatar: Taming Lifelike Audio-Driven Talking Head Generation with Comprehensive Motion and Lighting Control](https://mlsanity.com/paper/2412.19860/) (2024)
- [DLFR-VAE: Dynamic Latent Frame Rate VAE for Video Generation](https://mlsanity.com/paper/2502.11897/) (2025)

## Built on by (most cited first, 9 of 9)

- [Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspective](https://mlsanity.com/paper/2511.22249/) (2025)
- [Efficient Video Diffusion Models: Advancements and Challenges](https://mlsanity.com/paper/2604.15911/) (2026)
- [MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration](https://mlsanity.com/paper/2602.03028/) (2026)
- [Adaptive 1D Video Diffusion Autoencoder](https://mlsanity.com/paper/2602.04220/) (2026)
- [Latent-Compressed Variational Autoencoder for Video Diffusion Models](https://mlsanity.com/paper/2604.16479/) (2026)
- [PhiZero: A World Model Built Around Physical Language](https://mlsanity.com/paper/2607.28624/) (2026)
- [RiO-DETR: DETR for Real-time Oriented Object Detection](https://mlsanity.com/paper/2603.09411/) (2026)
- [TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization](https://mlsanity.com/paper/2606.17590/) (2026)
- [Latent Actions from Factorized Transition Effects under Agent Ambiguity](https://mlsanity.com/paper/2606.30544/) (2026)

## Abstract (arXiv)

First-frame-conditioned video autoencoders represent a clip with persistent content and a compact motion code. Although this removes much of the appearance redundancy, the remaining motion is typically compressed with a homogeneous latent geometry. Such representations use the same temporal support for broad scene evolution and fine-grained, frame-specific details. We introduce MotionStrata, which organizes a fixed motion budget into Global Motion and Detailed Motion. Temporally compressed Global queries summarize broad evolution, whereas frame-aligned Detailed queries preserve fine-grained structures whose configuration varies across frames. Frequency-guided routing and coarse-to-fine training establish this hierarchy without increasing motion dimensionality. Experiments show that MotionStrata maintains high reconstruction quality under aggressive compression and outperforms uniform and alternative grouped representations. Additional experiments evaluate hierarchical representation, downstream generation, and decoding cost. These results support hierarchical motion organization as a useful design principle for compact video autoencoding.