# Interactive Face Video Coding: A Generative Compression Framework

arXiv: https://arxiv.org/abs/2302.09919 · HTML: https://mlsanity.com/paper/2302.09919/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 135 references that are in the map.
- First posted on arXiv 20 Feb 2023.

## Builds on (most cited first, 30 of 55)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment](https://mlsanity.com/paper/2204.08958/) (2022)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering](https://mlsanity.com/paper/2109.08379/) (2021)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Depth-Aware Generative Adversarial Network for Talking Head Video Generation](https://mlsanity.com/paper/2203.06605/) (2022)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Morphable Face Models - An Open Framework](https://mlsanity.com/paper/1709.08398/) (2017)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)

## Built on by (most cited first, 17 of 17)

- [Generative Face Video Coding Techniques and Standardization Efforts: A Review](https://mlsanity.com/paper/2311.02649/) (2023)
- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [Geometric Prior Based Deep Human Point Cloud Geometry Compression](https://mlsanity.com/paper/2305.01309/) (2023)
- [Synchronous Multi-modal Semantic Communication System with Packet-level Coding](https://mlsanity.com/paper/2408.04535/) (2024)
- [Beyond GFVC: A Progressive Face Video Compression Framework with Adaptive Visual Tokens](https://mlsanity.com/paper/2410.08485/) (2024)
- [Generative Human Video Compression with Multi-granularity Temporal Trajectory Factorization](https://mlsanity.com/paper/2410.10171/) (2024)
- [Standardizing Generative Face Video Compression using Supplemental Enhancement Information](https://mlsanity.com/paper/2410.15105/) (2024)
- [Tokenizing Motion: A Generative Approach for Scene Dynamics Compression](https://mlsanity.com/paper/2410.09768/) (2024)
- [Pleno-Generation: A Scalable Generative Face Video Compression Framework with Bandwidth Intelligence](https://mlsanity.com/paper/2502.17085/) (2025)
- [Generative Models at the Frontier of Compression: A Survey on Generative Face Video Coding](https://mlsanity.com/paper/2506.07369/) (2025)
- [Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization](https://mlsanity.com/paper/2405.14221/) (2024)
- [Compressing Human Body Video with Interactive Semantics: A Generative Approach](https://mlsanity.com/paper/2505.16152/) (2025)
- [Rethinking Generative Human Video Coding with Implicit Motion Transformation](https://mlsanity.com/paper/2506.10453/) (2025)
- [Sparse2Dense: A Keypoint-driven Generative Framework for Human Video Compression and Vertex Prediction](https://mlsanity.com/paper/2509.23169/) (2025)
- [ExGS: Extreme 3D Gaussian Compression with Diffusion Priors](https://mlsanity.com/paper/2509.24758/) (2025)
- [A Lightweight Dual-Mode Optimization for Generative Face Video Coding](https://mlsanity.com/paper/2508.13547/) (2025)
- [Revising Second Order Terms in Deep Animation Video Coding](https://mlsanity.com/paper/2510.23561/) (2025)

## Abstract (arXiv)

In this paper, we propose a novel framework for Interactive Face Video Coding (IFVC), which allows humans to interact with the intrinsic visual representations instead of the signals. The proposed solution enjoys several distinct advantages, including ultra-compact representation, low delay interaction, and vivid expression/headpose animation. In particular, we propose the Internal Dimension Increase (IDI) based representation, greatly enhancing the fidelity and flexibility in rendering the appearance while maintaining reasonable representation cost. By leveraging strong statistical regularities, the visual signals can be effectively projected into controllable semantics in the three dimensional space (e.g., mouth motion, eye blinking, head rotation, head translation and head location), which are compressed and transmitted. The editable bitstream, which naturally supports the interactivity at the semantic level, can synthesize the face frames via the strong inference ability of the deep generative model. Experimental results have demonstrated the performance superiority and application prospects of our proposed IFVC scheme. In particular, the proposed scheme not only outperforms the state-of-the-art video coding standard Versatile Video Coding (VVC) and the latest generative compression schemes in terms of rate-distortion performance for face videos, but also enables the interactive coding without introducing additional manipulation processes. Furthermore, the proposed framework is expected to shed lights on the future design of the digital human communication in the metaverse.