mlsanityData to 14 Oct 2026

Large Generative Model-assisted Talking-face Semantic Communication System

arXiv 2411.03876 ↗First posted 6 Nov 2024cs.IT cs.LG math.ITcited by 3

Not on the map. Why?

Abstract (arXiv)

The rapid development of generative Artificial Intelligence (AI) continually unveils the potential of Semantic Communication (SemCom). However, current talking-face SemCom systems still encounter challenges such as low bandwidth utilization, semantic ambiguity, and diminished Quality of Experience (QoE). This study introduces a Large Generative Model-assisted Talking-face Semantic Communication (LGM-TSC) System tailored for the talking-face video communication. Firstly, we introduce a Generative Semantic Extractor (GSE) at the transmitter based on the FunASR model to convert semantically sparse talking-face videos into texts with high information density. Secondly, we establish a private Knowledge Base (KB) based on the Large Language Model (LLM) for semantic disambiguation and correction, complemented by a joint knowledge base-semantic-channel coding scheme. Finally, at the receiver, we propose a Generative Semantic Reconstructor (GSR) that utilizes BERT-VITS2 and SadTalker models to transform text back into a high-QoE talking-face video matching the user's timbre. Simulation results demonstrate the feasibility and effectiveness of the proposed LGM-TSC system.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 52 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

27 of 52 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  9. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  10. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  11. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 77
  12. 2021Nonlinear Transform Source-Channel Coding for Semantic Communicationscited by 76
15 more
  1. 2023Large Language Model Enhanced Multi-Agent Systems for 6G Communicationscited by 53
  2. 2022Deep Joint Source-Channel Coding for Semantic Communicationscited by 41
  3. 2023Large AI Model Empowered Multimodal Semantic Communicationscited by 38
  4. 2023Large AI Model-Based Semantic Communicationscited by 35
  5. 2022Deep Joint Source-Channel and Encryption Coding: Secure Semantic Communicationscited by 15
  6. 2024Visual Language Model based Cross-modal Semantic Communication Systemscited by 12
  7. 2021Txt2Vid: Ultra-Low Bitrate Compression of Talking-Head Videos via Textcited by 9
  8. 2023Towards Semantic Communication Protocols for 6G: From Protocol Learning to Language-Oriented Approachescited by 7
  9. 2024Semantic Importance-Aware Communications with Semantic Correction Using Large Language Modelscited by 6
  10. 2024Deep progressive reinforcement learning-based flexible resource scheduling framework for IRS and UAV-assisted MEC systemcited by 5
  11. 2024Video Semantic Communication with Major Object Extraction and Contextual Video Encodingcited by 3
  12. 2024Digital-Analog Transmission Framework for Task-Oriented Semantic Communicationscited by 3
  13. 2015Feedforward Sequential Memory Neural Networks without Recurrent Feedbackcited by 2
  14. 2023Deep Learning Enabled Semantic Communication Systems for Video Transmissioncited by 2
  15. 2024Rate Splitting Multiple Access-Enabled Adaptive Panoramic Video Semantic Transmissioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 20266G-Bench: An Open Benchmark for Semantic Communication and Network-Level Reasoning with Foundation Models in AI-Native 6G Networkscited by 7
  2. 2025Visual Fidelity Index for Generative Semantic Communications with Critical Information Embeddingcited by 3
  3. 2025From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communicationscited by 2