mlsanityData to 7 Oct 2026

A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation

arXiv 2304.00471 ↗First posted 2 Apr 2023cs.SD cs.CV cs.GRcited by 0

Not on the map. Why?

Abstract (arXiv)

Virtual humans have gained considerable attention in numerous industries, e.g., entertainment and e-commerce. As a core technology, synthesizing photorealistic face frames from target speech and facial identity has been actively studied with generative adversarial networks. Despite remarkable results of modern talking-face generation models, they often entail high computational burdens, which limit their efficient deployment. This study aims to develop a lightweight model for speech-driven talking-face synthesis. We build a compact generator by removing the residual blocks and reducing the channel width from Wav2Lip, a popular talking-face generator. We also present a knowledge distillation scheme to stably yet effectively train the small-capacity generator without adversarial learning. We reduce the number of parameters and MACs by 28× while retaining the performance of the original model. Moreover, to alleviate a severe performance drop when converting the whole generator to INT8 precision, we adopt a selective quantization method that uses FP16 for the quantization-sensitive layers and INT8 for the other layers. Using this mixed precision, we achieve up to a 19× speedup on edge GPUs without noticeably compromising the generation quality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 2 Apr 2023.

Builds on

19 of 25 references.

References · 19

  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  2. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  3. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  4. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  5. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  6. 2018Deep Audio-Visual Speech Recognitioncited by 318
  7. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  8. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  9. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  10. 2021Anycost GANs for Interactive Image Synthesis and Editingcited by 54
  11. 2020AutoGAN-Distiller: Searching to Compress Generative Adversarial Networkscited by 50
  12. 2021Content-Aware GAN Compressioncited by 28
7 more
  1. 2021Online Multi-Granularity Distillation for GAN Compressioncited by 24
  2. 2021Revisiting Discriminator in GAN Compression: A Generator-discriminator Cooperative Compression Schemecited by 23
  3. 2019QGAN: Quantized Generative Adversarial Networkscited by 20
  4. 2022Talking Face Generation with Multilingual TTScited by 18
  5. 2020Learning Efficient GANs for Image Translation via Differentiable Masks and co-Attention Distillationcited by 17
  6. 2022Cut Inner Layers: A Structured Pruning Strategy for Efficient U-Net GANscited by 5
  7. 2021Quantization of Generative Adversarial Networks for Efficient Inference: a Methodological Studycited by 4

Built on by

Not cited yet.