# Attention Is Not Always the Answer: Optimizing Voice Activity Detection with Simple Feature Fusion

arXiv: https://arxiv.org/abs/2506.01365 · HTML: https://mlsanity.com/paper/2506.01365/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Builds on (most cited first, 21 of 21)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [Spot the conversation: speaker diarisation in the wild](https://mlsanity.com/paper/2007.01216/) (2020)
- [UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training](https://mlsanity.com/paper/2110.05752/) (2021)
- [rVAD: An Unsupervised Segment-Based Robust Voice Activity Detection Method](https://mlsanity.com/paper/1906.03588/) (2019)
- [Multitask Detection of Speaker Changes, Overlapping Speech and Voice Activity Using wav2vec 2.0](https://mlsanity.com/paper/2210.14755/) (2022)
- [Profile-Error-Tolerant Target-Speaker Voice Activity Detection](https://mlsanity.com/paper/2309.12521/) (2023)
- [Investigating Prosodic Signatures via Speech Pre-Trained Models for Audio Deepfake Source Attribution](https://mlsanity.com/paper/2412.17796/) (2024)
- [Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization](https://mlsanity.com/paper/2412.19785/) (2024)
- [A Hybrid CNN-BiLSTM Voice Activity Detector](https://mlsanity.com/paper/2103.03529/) (2021)
- [Speaker Embeddings With Weakly Supervised Voice Activity Detection For Efficient Speaker Diarization](https://mlsanity.com/paper/2405.09142/) (2024)
- [A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech Enhancement](https://mlsanity.com/paper/2403.01369/) (2024)
- [Multi-View Multi-Task Modeling with Speech Foundation Models for Speech Forensic Tasks](https://mlsanity.com/paper/2410.12947/) (2024)

## Abstract (arXiv)

Voice Activity Detection (VAD) plays a key role in speech processing, often utilizing hand-crafted or neural features. This study examines the effectiveness of Mel-Frequency Cepstral Coefficients (MFCCs) and pre-trained model (PTM) features, including wav2vec 2.0, HuBERT, WavLM, UniSpeech, MMS, and Whisper. We propose FusionVAD, a unified framework that combines both feature types using three fusion strategies: concatenation, addition, and cross-attention (CA). Experimental results reveal that simple fusion techniques, particularly addition, outperform CA in both accuracy and efficiency. Fusion-based models consistently surpass single-feature models, highlighting the complementary nature of MFCCs and PTM features. Notably, our best-performing fusion model exceeds the state-of-the-art Pyannote across multiple datasets, achieving an absolute average improvement of 2.04%. These results confirm that simple feature fusion enhances VAD robustness while maintaining computational efficiency.