# Large Generative Model-assisted Talking-face Semantic Communication System

arXiv: https://arxiv.org/abs/2411.03876 · HTML: https://mlsanity.com/paper/2411.03876/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition](https://mlsanity.com/paper/1905.11235/) (2019)
- [Nonlinear Transform Source-Channel Coding for Semantic Communications](https://mlsanity.com/paper/2112.10961/) (2021)
- [Large Language Model Enhanced Multi-Agent Systems for 6G Communications](https://mlsanity.com/paper/2312.07850/) (2023)
- [Deep Joint Source-Channel Coding for Semantic Communications](https://mlsanity.com/paper/2211.08747/) (2022)
- [Large AI Model Empowered Multimodal Semantic Communications](https://mlsanity.com/paper/2309.01249/) (2023)
- [Large AI Model-Based Semantic Communications](https://mlsanity.com/paper/2307.03492/) (2023)
- [Deep Joint Source-Channel and Encryption Coding: Secure Semantic Communications](https://mlsanity.com/paper/2208.09245/) (2022)
- [Visual Language Model based Cross-modal Semantic Communication Systems](https://mlsanity.com/paper/2407.00020/) (2024)
- [Txt2Vid: Ultra-Low Bitrate Compression of Talking-Head Videos via Text](https://mlsanity.com/paper/2106.14014/) (2021)
- [Towards Semantic Communication Protocols for 6G: From Protocol Learning to Language-Oriented Approaches](https://mlsanity.com/paper/2310.09506/) (2023)
- [Semantic Importance-Aware Communications with Semantic Correction Using Large Language Models](https://mlsanity.com/paper/2405.16011/) (2024)
- [Deep progressive reinforcement learning-based flexible resource scheduling framework for IRS and UAV-assisted MEC system](https://mlsanity.com/paper/2408.01248/) (2024)
- [Video Semantic Communication with Major Object Extraction and Contextual Video Encoding](https://mlsanity.com/paper/2402.01330/) (2024)
- [Digital-Analog Transmission Framework for Task-Oriented Semantic Communications](https://mlsanity.com/paper/2407.11350/) (2024)
- [Feedforward Sequential Memory Neural Networks without Recurrent Feedback](https://mlsanity.com/paper/1510.02693/) (2015)
- [Deep Learning Enabled Semantic Communication Systems for Video Transmission](https://mlsanity.com/paper/2312.05062/) (2023)
- [Rate Splitting Multiple Access-Enabled Adaptive Panoramic Video Semantic Transmission](https://mlsanity.com/paper/2402.16581/) (2024)

## Built on by (most cited first, 3 of 3)

- [6G-Bench: An Open Benchmark for Semantic Communication and Network-Level Reasoning with Foundation Models in AI-Native 6G Networks](https://mlsanity.com/paper/2602.08675/) (2026)
- [Visual Fidelity Index for Generative Semantic Communications with Critical Information Embedding](https://mlsanity.com/paper/2505.10405/) (2025)
- [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025)

## Abstract (arXiv)

The rapid development of generative Artificial Intelligence (AI) continually unveils the potential of Semantic Communication (SemCom). However, current talking-face SemCom systems still encounter challenges such as low bandwidth utilization, semantic ambiguity, and diminished Quality of Experience (QoE). This study introduces a Large Generative Model-assisted Talking-face Semantic Communication (LGM-TSC) System tailored for the talking-face video communication. Firstly, we introduce a Generative Semantic Extractor (GSE) at the transmitter based on the FunASR model to convert semantically sparse talking-face videos into texts with high information density. Secondly, we establish a private Knowledge Base (KB) based on the Large Language Model (LLM) for semantic disambiguation and correction, complemented by a joint knowledge base-semantic-channel coding scheme. Finally, at the receiver, we propose a Generative Semantic Reconstructor (GSR) that utilizes BERT-VITS2 and SadTalker models to transform text back into a high-QoE talking-face video matching the user's timbre. Simulation results demonstrate the feasibility and effectiveness of the proposed LGM-TSC system.