# Modality Dropout for Improved Performance-driven Talking Faces

arXiv: https://arxiv.org/abs/2005.13616 · HTML: https://mlsanity.com/paper/2005.13616/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2020.

## Built on by (most cited first, 30 of 32)

- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization](https://mlsanity.com/paper/2106.04185/) (2021)
- [DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation](https://mlsanity.com/paper/2401.04747/) (2024)
- [Dyadic Interaction Modeling for Social Behavior Generation](https://mlsanity.com/paper/2403.09069/) (2024)
- [On-the-fly Modulation for Balanced Multimodal Learning](https://mlsanity.com/paper/2410.11582/) (2024)
- [Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation](https://mlsanity.com/paper/2310.03986/) (2023)
- [LaughTalk: Expressive 3D Talking Head Generation with Laughter](https://mlsanity.com/paper/2311.00994/) (2023)
- [Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2304.10756/) (2023)
- [Language-Guided Music Recommendation for Video via Prompt Analogies](https://mlsanity.com/paper/2306.09327/) (2023)
- [Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videos](https://mlsanity.com/paper/2203.03014/) (2022)
- [Promoting cross-modal representations to improve multimodal foundation models for physiological signals](https://mlsanity.com/paper/2410.16424/) (2024)
- [MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection](https://mlsanity.com/paper/2410.03010/) (2024)
- [Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation](https://mlsanity.com/paper/2212.10313/) (2022)
- [Audio-Driven 3D Facial Animation from In-the-Wild Videos](https://mlsanity.com/paper/2306.11541/) (2023)
- [Learning Contrastive Multimodal Fusion with Improved Modality Dropout for Disease Detection and Prediction](https://mlsanity.com/paper/2509.18284/) (2025)
- [What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?](https://mlsanity.com/paper/2310.06383/) (2023)
- [AVFace: Towards Detailed Audio-Visual 4D Face Reconstruction](https://mlsanity.com/paper/2304.13115/) (2023)
- [Modality Dropout for Multimodal Device Directed Speech Detection using Verbal and Non-Verbal Features](https://mlsanity.com/paper/2310.15261/) (2023)
- [Multimodal Negative Learning](https://mlsanity.com/paper/2510.20877/) (2025)
- [Models Designed to Forget: Machine Unlearning via Key Deletion](https://mlsanity.com/paper/2603.15033/) (2026)
- [On the role of Lip Articulation in Visual Speech Perception](https://mlsanity.com/paper/2203.10117/) (2022)
- [x2-Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space](https://mlsanity.com/paper/2603.16671/) (2026)
- [Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention](https://mlsanity.com/paper/2302.12532/) (2023)
- [AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars](https://mlsanity.com/paper/2505.15058/) (2025)
- [FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs](https://mlsanity.com/paper/2509.11425/) (2025)
- [Towards Unified Co-Speech Gesture Generation via Hierarchical Implicit Periodicity Learning](https://mlsanity.com/paper/2512.13131/) (2025)
- [MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion](https://mlsanity.com/paper/2604.02941/) (2026)
- [DS@GT ARC at MEDIQA-CORE-Task-1 2026: Trimodal Model Fusion with Task-Specific Gates for Brain Tumor Subtype Classification](https://mlsanity.com/paper/2608.00086/) (2026)

## Abstract (arXiv)

We describe our novel deep learning approach for driving animated faces using both acoustic and visual information. In particular, speech-related facial movements are generated using audiovisual information, and non-speech facial movements are generated using only visual information. To ensure that our model exploits both modalities during training, batches are generated that contain audio-only, video-only, and audiovisual input features. The probability of dropping a modality allows control over the degree to which the model exploits audio and visual information during training. Our trained model runs in real-time on resource limited hardware (e.g. a smart phone), it is user agnostic, and it is not dependent on a potentially error-prone transcription of the speech. We use subjective testing to demonstrate: 1) the improvement of audiovisual-driven animation over the equivalent video-only approach, and 2) the improvement in the animation of speech-related facial movements after introducing modality dropout. Before introducing dropout, viewers prefer audiovisual-driven animation in 51% of the test sequences compared with only 18% for video-driven. After introducing dropout viewer preference for audiovisual-driven animation increases to 74%, but decreases to 8% for video-only.