# Modality Dropout for Multimodal Device Directed Speech Detection using Verbal and Non-Verbal Features

arXiv: https://arxiv.org/abs/2310.15261 · HTML: https://mlsanity.com/paper/2310.15261/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Modality Dropout for Improved Performance-driven Talking Faces](https://mlsanity.com/paper/2005.13616/) (2020)
- [Device-directed Utterance Detection](https://mlsanity.com/paper/1808.02504/) (2018)
- [Unacceptable, where is my privacy? Exploring Accidental Triggers of Smart Speakers](https://mlsanity.com/paper/2008.00508/) (2020)
- [Streaming Transformer for Hardware Efficient Voice Trigger Detection and False Trigger Mitigation](https://mlsanity.com/paper/2105.06598/) (2021)
- [Lattice-based Improvements for Voice Triggering Using Graph Neural Networks](https://mlsanity.com/paper/2001.10822/) (2020)
- [Voice trigger detection from LVCSR hypothesis lattices using bidirectional lattice recurrent neural networks](https://mlsanity.com/paper/2003.00304/) (2020)
- [Streaming on-device detection of device directed speech from voice and touch-based invocation](https://mlsanity.com/paper/2110.04656/) (2021)

## Built on by (most cited first, 4 of 4)

- [SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions](https://mlsanity.com/paper/2501.19377/) (2025)
- [Learning Contrastive Multimodal Fusion with Improved Modality Dropout for Disease Detection and Prediction](https://mlsanity.com/paper/2509.18284/) (2025)
- [Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI](https://mlsanity.com/paper/2604.08412/) (2026)
- [Less can be More: What Aspects of Speech Drive End-of-Turn Detection](https://mlsanity.com/paper/2609.11066/) (2026)

## Abstract (arXiv)

Device-directed speech detection (DDSD) is the binary classification task of distinguishing between queries directed at a voice assistant versus side conversation or background speech. State-of-the-art DDSD systems use verbal cues, e.g acoustic, text and/or automatic speech recognition system (ASR) features, to classify speech as device-directed or otherwise, and often have to contend with one or more of these modalities being unavailable when deployed in real-world settings. In this paper, we investigate fusion schemes for DDSD systems that can be made more robust to missing modalities. Concurrently, we study the use of non-verbal cues, specifically prosody features, in addition to verbal cues for DDSD. We present different approaches to combine scores and embeddings from prosody with the corresponding verbal cues, finding that prosody improves DDSD performance by upto 8.5% in terms of false acceptance rate (FA) at a given fixed operating point via non-linear intermediate fusion, while our use of modality dropout techniques improves the performance of these models by 7.4% in terms of FA when evaluated with missing modalities during inference time.