# Attentive Merging of Hidden Embeddings from Pre-trained Speech Model for Anti-spoofing Detection

arXiv: https://arxiv.org/abs/2406.10283 · HTML: https://mlsanity.com/paper/2406.10283/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [End-to-end anti-spoofing with RawNet2](https://mlsanity.com/paper/2011.01108/) (2020)
- [ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection](https://mlsanity.com/paper/2109.00537/) (2021)
- [Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation](https://mlsanity.com/paper/2202.12233/) (2022)
- [Exploring wav2vec 2.0 on speaker verification and language identification](https://mlsanity.com/paper/2012.06185/) (2020)
- [ASVspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech](https://mlsanity.com/paper/2102.05889/) (2021)
- [Investigating self-supervised front ends for speech spoofing countermeasures](https://mlsanity.com/paper/2111.07725/) (2021)
- [The Vicomtech Audio Deepfake Detection System based on Wav2Vec2 for the 2022 ADD Challenge](https://mlsanity.com/paper/2203.01573/) (2022)
- [Deep Residual Neural Networks for Audio Spoofing Detection](https://mlsanity.com/paper/1907.00501/) (2019)
- [ASSERT: Anti-Spoofing with Squeeze-Excitation and Residual neTworks](https://mlsanity.com/paper/1904.01120/) (2019)
- [Disentangling Voice and Content with Self-Supervision for Speaker Recognition](https://mlsanity.com/paper/2310.01128/) (2023)
- [Fully Automated End-to-End Fake Audio Detection](https://mlsanity.com/paper/2208.09618/) (2022)
- [UR Channel-Robust Synthetic Speech Detection System for ASVspoof 2021](https://mlsanity.com/paper/2107.12018/) (2021)

## Built on by (most cited first, 17 of 17)

- [DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Dataset](https://mlsanity.com/paper/2409.08731/) (2024)
- [Mixture of Experts Fusion for Fake Audio Detection Using Frozen wav2vec 2.0](https://mlsanity.com/paper/2409.11909/) (2024)
- [Nes2Net: A Lightweight Nested Architecture for Foundation Model Driven Speech Anti-spoofing](https://mlsanity.com/paper/2504.05657/) (2025)
- [Speech Foundation Model Ensembles for the Controlled Singing Voice Deepfake Detection (CtrSVDD) Challenge 2024](https://mlsanity.com/paper/2409.02302/) (2024)
- [Towards Quantifying and Reducing Language Mismatch Effects in Cross-Lingual Speech Anti-Spoofing](https://mlsanity.com/paper/2409.08346/) (2024)
- [Detect All-Type Deepfake Audio: Wavelet Prompt Tuning for Enhanced Auditory Perception](https://mlsanity.com/paper/2504.06753/) (2025)
- [Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox](https://mlsanity.com/paper/2605.27772/) (2026)
- [Interpretable All-Type Audio Deepfake Detection with Audio LLMs via Frequency-Time Reinforcement Learning](https://mlsanity.com/paper/2601.02983/) (2026)
- [Passive Deepfake Detection Across Multi-modalities: A Comprehensive Survey](https://mlsanity.com/paper/2411.17911/) (2024)
- [AT-ADD: All-Type Audio Deepfake Detection Challenge Evaluation Plan](https://mlsanity.com/paper/2604.08184/) (2026)
- [Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization](https://mlsanity.com/paper/2502.12672/) (2025)
- [QAMO: Quality-aware Multi-centroid One-class Learning For Speech Deepfake Detection](https://mlsanity.com/paper/2509.20679/) (2025)
- [Generalizable Speech Deepfake Detection via Information Bottleneck Enhanced Adversarial Alignment](https://mlsanity.com/paper/2509.23618/) (2025)
- [Quantizer-Aware Hierarchical Neural Codec Modeling for Speech Deepfake Detection](https://mlsanity.com/paper/2603.16914/) (2026)
- [Dual-Granularity Orthogonal Disentanglement for Generalizable Audio Deepfake Detection](https://mlsanity.com/paper/2606.16532/) (2026)
- [Supervised Post-training of Speech Foundation Models for Robust Adaptation in Speech Deepfake Detection](https://mlsanity.com/paper/2606.25328/) (2026)
- [Learning as Deepfakes Evolve: RF-Prompt for Continual Audio Deepfake Detection](https://mlsanity.com/paper/2609.37586/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) speech representation models, trained on large speech corpora, have demonstrated effectiveness in extracting hierarchical speech embeddings through multiple transformer layers. However, the behavior of these embeddings in specific tasks remains uncertain. This paper investigates the multi-layer behavior of the WavLM model in anti-spoofing and proposes an attentive merging method to leverage the hierarchical hidden embeddings. Results demonstrate the feasibility of fine-tuning WavLM to achieve the best equal error rate (EER) of 0.65%, 3.50%, and 3.19% on the ASVspoof 2019LA, 2021LA, and 2021DF evaluation sets, respectively. Notably, We find that the early hidden transformer layers of the WavLM large model contribute significantly to anti-spoofing task, enabling computational efficiency by utilizing a partial pre-trained model.