# Latent Video Dataset Distillation

arXiv: https://arxiv.org/abs/2504.17132 · HTML: https://mlsanity.com/paper/2504.17132/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 30 of 32)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Dataset Condensation with Distribution Matching](https://mlsanity.com/paper/2110.04181/) (2021)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [Dataset Meta-Learning from Kernel Ridge-Regression](https://mlsanity.com/paper/2011.00050/) (2020)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory](https://mlsanity.com/paper/2211.10586/) (2022)
- [Dataset Distillation using Neural Feature Regression](https://mlsanity.com/paper/2206.00719/) (2022)
- [Dataset Pruning: Reducing Training Data by Examining Generalization Influence](https://mlsanity.com/paper/2205.09329/) (2022)
- [Efficient Dataset Distillation Using Random Feature Approximation](https://mlsanity.com/paper/2210.12067/) (2022)
- [DREAM: Efficient Dataset Distillation by Representative Matching](https://mlsanity.com/paper/2302.14416/) (2023)
- [CV-VAE: A Compatible Video VAE for Latent Generative Video Models](https://mlsanity.com/paper/2405.20279/) (2024)
- [DC-BENCH: Dataset Condensation Benchmark](https://mlsanity.com/paper/2207.09639/) (2022)
- [DiM: Distilling Dataset into Generative Model](https://mlsanity.com/paper/2303.04707/) (2023)
- [No frame left behind: Full Video Action Recognition](https://mlsanity.com/paper/2103.15395/) (2021)
- [Unlocking Dataset Distillation with Diffusion Models](https://mlsanity.com/paper/2403.03881/) (2024)
- [I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models](https://mlsanity.com/paper/2405.17849/) (2024)
- [HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator](https://mlsanity.com/paper/2209.07143/) (2022)
- [Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement](https://mlsanity.com/paper/2312.00362/) (2023)
- [Generative Dataset Distillation: Balancing Global Structure and Local Details](https://mlsanity.com/paper/2404.17732/) (2024)
- [Dataset Distillation in Latent Space](https://mlsanity.com/paper/2311.15547/) (2023)
- [Video Set Distillation: Information Diversification and Temporal Densification](https://mlsanity.com/paper/2412.00111/) (2024)

## Built on by (most cited first, 1 of 1)

- [VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstruction](https://mlsanity.com/paper/2511.18831/) (2025)

## Abstract (arXiv)

Dataset distillation has demonstrated remarkable effectiveness in high-compression scenarios for image datasets. While video datasets inherently contain greater redundancy, existing video dataset distillation methods primarily focus on compression in the pixel space, overlooking advances in the latent space that have been widely adopted in modern text-to-image and text-to-video models. In this work, we bridge this gap by introducing a novel video dataset distillation approach that operates in the latent space using a state-of-the-art variational encoder. Furthermore, we employ a diversity-aware data selection strategy to select both representative and diverse samples. Additionally, we introduce a simple, training-free method to further compress the distilled latent dataset. By combining these techniques, our approach achieves a new state-of-the-art performance in dataset distillation, outperforming prior methods on all datasets, e.g. on HMDB51 IPC 1, we achieve a 2.6% performance increase; on MiniUCF IPC 5, we achieve a 7.8% performance increase. Our code is available at https://github.com/liningresearch/Latent_Video_Dataset_Distillation.