# Sparse2Dense: A Keypoint-driven Generative Framework for Human Video Compression and Vertex Prediction

arXiv: https://arxiv.org/abs/2509.23169 · HTML: https://mlsanity.com/paper/2509.23169/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Builds on (most cited first, 24 of 24)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](https://mlsanity.com/paper/2311.16498/) (2023)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Thin-Plate Spline Motion Model for Image Animation](https://mlsanity.com/paper/2203.14367/) (2022)
- [Latent Image Animator: Learning to Animate Images via Latent Space Navigation](https://mlsanity.com/paper/2203.09043/) (2022)
- [One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer](https://mlsanity.com/paper/2303.16160/) (2023)
- [Predicting 3D Human Dynamics from Video](https://mlsanity.com/paper/1908.04781/) (2019)
- [Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications](https://mlsanity.com/paper/2008.02793/) (2020)
- [Generative Face Video Coding Techniques and Standardization Efforts: A Review](https://mlsanity.com/paper/2311.02649/) (2023)
- [Interactive Face Video Coding: A Generative Compression Framework](https://mlsanity.com/paper/2302.09919/) (2023)
- [Perceptual Quality Assessment of Face Video Compression: A Benchmark and An Effective Method](https://mlsanity.com/paper/2304.07056/) (2023)
- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [Generative Human Video Compression with Multi-granularity Temporal Trajectory Factorization](https://mlsanity.com/paper/2410.10171/) (2024)
- [Representation learning of vertex heatmaps for 3D human mesh reconstruction from multi-view images](https://mlsanity.com/paper/2306.16615/) (2023)
- [Standardizing Generative Face Video Compression using Supplemental Enhancement Information](https://mlsanity.com/paper/2410.15105/) (2024)
- [Generative Models at the Frontier of Compression: A Survey on Generative Face Video Coding](https://mlsanity.com/paper/2506.07369/) (2025)
- [Compressing Human Body Video with Interactive Semantics: A Generative Approach](https://mlsanity.com/paper/2505.16152/) (2025)
- [Rethinking Generative Human Video Coding with Implicit Motion Transformation](https://mlsanity.com/paper/2506.10453/) (2025)

## Built on by (most cited first, 1 of 1)

- [Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents](https://mlsanity.com/paper/2605.20061/) (2026)

## Abstract (arXiv)

For bandwidth-constrained multimedia applications, simultaneously achieving ultra-low bitrate human video compression and accurate vertex prediction remains a critical challenge, as it demands the harmonization of dynamic motion modeling, detailed appearance synthesis, and geometric consistency. To address this challenge, we propose Sparse2Dense, a keypoint-driven generative framework that leverages extremely sparse 3D keypoints as compact transmitted symbols to enable ultra-low bitrate human video compression and precise human vertex prediction. The key innovation is the multi-task learning-based and keypoint-aware deep generative model, which could encode complex human motion via compact 3D keypoints and leverage these sparse keypoints to estimate dense motion for video synthesis with temporal coherence and realistic textures. Additionally, a vertex predictor is integrated to learn human vertex geometry through joint optimization with video generation, ensuring alignment between visual content and geometric structure. Extensive experiments demonstrate that the proposed Sparse2Dense framework achieves competitive compression performance for human video over traditional/generative video codecs, whilst enabling precise human vertex prediction for downstream geometry applications. As such, Sparse2Dense is expected to facilitate bandwidth-efficient human-centric media transmission, such as real-time motion analysis, virtual human animation, and immersive entertainment.