# Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI

arXiv: https://arxiv.org/abs/2604.08412 · HTML: https://mlsanity.com/paper/2604.08412/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Builds on (most cited first, 13 of 13)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Device-directed Utterance Detection](https://mlsanity.com/paper/1808.02504/) (2018)
- [SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions](https://mlsanity.com/paper/2501.19377/) (2025)
- [QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge](https://mlsanity.com/paper/2306.17404/) (2023)
- [Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection](https://mlsanity.com/paper/2406.09617/) (2024)
- [Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models](https://mlsanity.com/paper/2312.03632/) (2023)
- [A Multimodal Approach to Device-Directed Speech Detection with Large Language Models](https://mlsanity.com/paper/2403.14438/) (2024)
- [Streaming on-device detection of device directed speech from voice and touch-based invocation](https://mlsanity.com/paper/2110.04656/) (2021)
- [Modality Dropout for Multimodal Device Directed Speech Detection using Verbal and Non-Verbal Features](https://mlsanity.com/paper/2310.15261/) (2023)
- [Adaptive Knowledge Distillation for Device-Directed Speech Detection](https://mlsanity.com/paper/2508.02801/) (2025)
- [Implicit Acoustic Echo Cancellation for Keyword Spotting and Device-Directed Speech Detection](https://mlsanity.com/paper/2111.10639/) (2021)
- [Device-Directed Speech Detection for Follow-up Conversations Using Large Language Models](https://mlsanity.com/paper/2411.00023/) (2024)
- [PCIE_Interaction Solution for Ego4D Social Interaction Challenge](https://mlsanity.com/paper/2505.24404/) (2025)

## Abstract (arXiv)

We study device-addressed speech detection under pre-ASR edge deployment constraints, where systems must decide whether to forward audio before transcription under strict latency and compute limits. We show that, in multi-speaker environments with temporally ambiguous utterances, this task is more effectively modelled as a sequential routing problem over interaction history than as an utterance-local classification task. We formalize this as Sequential Device-Addressed Routing (SDAR) and present the Selective Attention System (SAS), an on-device implementation that instantiates this formulation. On a held-out 60-hour multi-speaker English test set, the primary audio-only configuration achieves F1=0.86 (precision=0.89, recall=0.83); with an optional camera, audio+video fusion raises F1 to 0.95 (precision=0.97, recall=0.93). Removing causal interaction history (Stage~3) reduced F1 from 0.95 to 0.57+/-0.03 in the audio+video configuration under our evaluation protocol. Among the tested components, this was the largest observed ablation effect, indicating that short-horizon interaction history carries substantial decision-relevant information in the evaluated setting. SAS runs fully on-device on ARM Cortex-A class hardware (<150 ms latency, <20 MB footprint). All results are from internal evaluation on a proprietary dataset evaluated primarily in English; a 5-hour evaluation subset may be shared for independent verification (Section 8.8).