# A Realistic Face-to-Face Conversation System based on Deep Neural Networks

arXiv: https://arxiv.org/abs/1908.07750 · HTML: https://mlsanity.com/paper/1908.07750/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2019.

## Built on by (most cited first, 6 of 6)

- [Didn't see that coming: a survey on non-verbal social human behavior forecasting](https://mlsanity.com/paper/2203.02480/) (2022)
- [AgentAvatar: Disentangling Planning, Driving and Rendering for Photorealistic Avatar Agents](https://mlsanity.com/paper/2311.17465/) (2023)
- [Towards Unbiased Label Distribution Learning for Facial Pose Estimation Using Anisotropic Spherical Gaussian](https://mlsanity.com/paper/2208.09122/) (2022)
- [REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2306.06583/) (2023)
- [How Self-Supervised Learning Can be Used for Fine-Grained Head Pose Estimation?](https://mlsanity.com/paper/2108.04893/) (2021)
- [A Waste Copper Granules Rating System Based on Machine Vision](https://mlsanity.com/paper/2207.04575/) (2022)

## Abstract (arXiv)

To improve the experiences of face-to-face conversation with avatar, this paper presents a novel conversation system. It is composed of two sequence-to-sequence models respectively for listening and speaking and a Generative Adversarial Network (GAN) based realistic avatar synthesizer. The models exploit the facial action and head pose to learn natural human reactions. Based on the models' output, the synthesizer uses the Pixel2Pixel model to generate realistic facial images. To show the improvement of our system, we use a 3D model based avatar driving scheme as a reference. We train and evaluate our neural networks with the data from ESPN shows. Experimental results show that our conversation system can generate natural facial reactions and realistic facial images.