# Interactive Conversational Head Generation

arXiv: https://arxiv.org/abs/2307.02090 · HTML: https://mlsanity.com/paper/2307.02090/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 62 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 24 of 24)

- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering](https://mlsanity.com/paper/2109.08379/) (2021)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation](https://mlsanity.com/paper/2301.03786/) (2023)
- [MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement](https://mlsanity.com/paper/1810.08768/) (2018)
- [You said that?](https://mlsanity.com/paper/1705.02966/) (2017)
- [Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis](https://mlsanity.com/paper/2207.11770/) (2022)
- [FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning](https://mlsanity.com/paper/2108.07938/) (2021)
- [Recycle-GAN: Unsupervised Video Retargeting](https://mlsanity.com/paper/1808.05174/) (2018)
- [Deep Audio-Visual Learning: A Survey](https://mlsanity.com/paper/2001.04758/) (2020)
- [Responsive Listening Head Generation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2112.13548/) (2021)
- [On Learning 3D Face Morphable Model from In-the-wild Images](https://mlsanity.com/paper/1808.09560/) (2018)
- [Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction](https://mlsanity.com/paper/1906.04158/) (2019)
- [The JDDC Corpus: A Large-Scale Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Service](https://mlsanity.com/paper/1911.09969/) (2019)
- [Speech Driven Talking Face Generation from a Single Image and an Emotion Condition](https://mlsanity.com/paper/2008.03592/) (2020)
- [Deep-FSMN for Large Vocabulary Continuous Speech Recognition](https://mlsanity.com/paper/1803.05030/) (2018)
- [Free-HeadGAN: Neural Talking Head Synthesis with Explicit Gaze Control](https://mlsanity.com/paper/2208.02210/) (2022)
- [A Temporally-Aware Interpolation Network for Video Frame Inpainting](https://mlsanity.com/paper/1803.07218/) (2018)

## Built on by (most cited first, 14 of 14)

- [INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations](https://mlsanity.com/paper/2412.04037/) (2024)
- [X-Streamer: Unified Human World Modeling with Audiovisual Interaction](https://mlsanity.com/paper/2509.21574/) (2025)
- [ARIG: Autoregressive Interactive Head Generation for Real-time Conversations](https://mlsanity.com/paper/2507.00472/) (2025)
- [DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion](https://mlsanity.com/paper/2504.04010/) (2025)
- [DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Model](https://mlsanity.com/paper/2512.24408/) (2025)
- [Hierarchical Semantic Perceptual Listener Head Video Generation: A High-performance Pipeline](https://mlsanity.com/paper/2307.09821/) (2023)
- [Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamics](https://mlsanity.com/paper/2512.15340/) (2025)
- [Multi-human Interactive Talking Dataset](https://mlsanity.com/paper/2508.03050/) (2025)
- [Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencing](https://mlsanity.com/paper/2510.03548/) (2025)
- [Learning and Evaluating Human Preferences for Conversational Head Generation](https://mlsanity.com/paper/2307.10636/) (2023)
- [TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation](https://mlsanity.com/paper/2512.20296/) (2025)
- [ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation](https://mlsanity.com/paper/2603.17427/) (2026)
- [MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations](https://mlsanity.com/paper/2606.27779/) (2026)
- [EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold](https://mlsanity.com/paper/2609.35616/) (2026)

## Abstract (arXiv)

We introduce a new conversation head generation benchmark for synthesizing behaviors of a single interlocutor in a face-to-face conversation. The capability to automatically synthesize interlocutors which can participate in long and multi-turn conversations is vital and offer benefits for various applications, including digital humans, virtual agents, and social robots. While existing research primarily focuses on talking head generation (one-way interaction), hindering the ability to create a digital human for conversation (two-way) interaction due to the absence of listening and interaction parts. In this work, we construct two datasets to address this issue, ``ViCo'' for independent talking and listening head generation tasks at the sentence level, and ``ViCo-X'', for synthesizing interlocutors in multi-turn conversational scenarios. Based on ViCo and ViCo-X, we define three novel tasks targeting the interaction modeling during the face-to-face conversation: 1) responsive listening head generation making listeners respond actively to the speaker with non-verbal signals, 2) expressive talking head generation guiding speakers to be aware of listeners' behaviors, and 3) conversational head generation to integrate the talking/listening ability in one interlocutor. Along with the datasets, we also propose corresponding baseline solutions to the three aforementioned tasks. Experimental results show that our baseline method could generate responsive and vivid agents that can collaborate with real person to fulfil the whole conversation. Project page: https://vico.solutions/.