# FaceChat: An Emotion-Aware Face-to-face Dialogue Framework

arXiv: https://arxiv.org/abs/2303.07316 · HTML: https://mlsanity.com/paper/2303.07316/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog](https://mlsanity.com/paper/1902.00579/) (2019)
- [Sentiment Adaptive End-to-End Dialog Systems](https://mlsanity.com/paper/1804.10731/) (2018)
- [Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System](https://mlsanity.com/paper/2205.04006/) (2022)

## Built on by (most cited first, 3 of 3)

- [Fairness of ChatGPT and the Role Of Explainable-Guided Prompts](https://mlsanity.com/paper/2307.11761/) (2023)
- [AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues](https://mlsanity.com/paper/2412.17292/) (2024)
- [Dialogue Systems Can Generate Appropriate Responses without the Use of Question Marks? -- Investigation of the Effects of Question Marks on Dialogue Systems](https://mlsanity.com/paper/2308.03293/) (2023)

## Abstract (arXiv)

While current dialogue systems like ChatGPT have made significant advancements in text-based interactions, they often overlook the potential of other modalities in enhancing the overall user experience. We present FaceChat, a web-based dialogue framework that enables emotionally-sensitive and face-to-face conversations. By seamlessly integrating cutting-edge technologies in natural language processing, computer vision, and speech processing, FaceChat delivers a highly immersive and engaging user experience. FaceChat framework has a wide range of potential applications, including counseling, emotional support, and personalized customer service. The system is designed to be simple and flexible as a platform for future researchers to advance the field of multimodal dialogue systems. The code is publicly available at https://github.com/qywu/FaceChat.