# Temporal-Channel Modeling in Multi-head Self-Attention for Synthetic Speech Detection

arXiv: https://arxiv.org/abs/2406.17376 · HTML: https://mlsanity.com/paper/2406.17376/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [End-to-end anti-spoofing with RawNet2](https://mlsanity.com/paper/2011.01108/) (2020)
- [Does Audio Deepfake Detection Generalize?](https://mlsanity.com/paper/2203.16263/) (2022)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild](https://mlsanity.com/paper/2210.02437/) (2022)
- [Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation](https://mlsanity.com/paper/2202.12233/) (2022)
- [RawBoost: A Raw Data Boosting and Augmentation Method applied to Automatic Speaker Verification Anti-Spoofing](https://mlsanity.com/paper/2111.04433/) (2021)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [STC Antispoofing Systems for the ASVspoof2019 Challenge](https://mlsanity.com/paper/1904.05576/) (2019)
- [Replay and Synthetic Speech Detection with Res2net Architecture](https://mlsanity.com/paper/2010.15006/) (2020)
- [Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets](https://mlsanity.com/paper/2210.05958/) (2022)
- [Light Convolutional Neural Network with Feature Genuinization for Detection of Synthetic Speech Attacks](https://mlsanity.com/paper/2009.09637/) (2020)
- [Channel-wise Gated Res2Net: Towards Robust Detection of Synthetic Speech Attacks](https://mlsanity.com/paper/2107.08803/) (2021)
- [An explainability study of the constant Q cepstral coefficient spoofing countermeasure for automatic speaker verification](https://mlsanity.com/paper/2004.06422/) (2020)
- [The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCF](https://mlsanity.com/paper/1304.2865/) (2013)
- [The defender's perspective on automatic speaker verification: An overview](https://mlsanity.com/paper/2305.12804/) (2023)
- [HM-Conformer: A Conformer-based audio deepfake detection system with hierarchical pooling and multi-level classification token aggregation methods](https://mlsanity.com/paper/2309.08208/) (2023)
- [Estimation of speaker age and height from speech signal using bi-encoder transformer mixture model](https://mlsanity.com/paper/2203.11774/) (2022)
- [Voice Spoofing Countermeasures: Taxonomy, State-of-the-art, experimental analysis of generalizability, open challenges, and the way forward](https://mlsanity.com/paper/2210.00417/) (2022)
- [Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verification](https://mlsanity.com/paper/2309.14838/) (2023)
- [Efficient Attention Branch Network with Combined Loss Function for Automatic Speaker Verification Spoof Detection](https://mlsanity.com/paper/2109.02051/) (2021)

## Built on by (most cited first, 25 of 25)

- [XLSR-Mamba: A Dual-Column Bidirectional State Space Model for Spoofing Attack Detection](https://mlsanity.com/paper/2411.10027/) (2024)
- [Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models](https://mlsanity.com/paper/2509.02859/) (2025)
- [Nes2Net: A Lightweight Nested Architecture for Foundation Model Driven Speech Anti-spoofing](https://mlsanity.com/paper/2504.05657/) (2025)
- [Speech Foundation Model Ensembles for the Controlled Singing Voice Deepfake Detection (CtrSVDD) Challenge 2024](https://mlsanity.com/paper/2409.02302/) (2024)
- [Replay Attacks Against Audio Deepfake Detection](https://mlsanity.com/paper/2505.14862/) (2025)
- [Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative](https://mlsanity.com/paper/2508.09294/) (2025)
- [Unveiling Audio Deepfake Origins: A Deep Metric learning And Conformer Network Approach With Ensemble Fusion](https://mlsanity.com/paper/2506.02085/) (2025)
- [Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR](https://mlsanity.com/paper/2603.06164/) (2026)
- [Bona fide Cross Testing Reveals Weak Spot in Audio Deepfake Detection Systems](https://mlsanity.com/paper/2509.09204/) (2025)
- [Pushing the Performance of Synthetic Speech Detection with Kolmogorov-Arnold Networks and Self-Supervised Learning Models](https://mlsanity.com/paper/2506.14153/) (2025)
- [Addressing Gradient Misalignment in Data-Augmented Training for Robust Speech Deepfake Detection](https://mlsanity.com/paper/2509.20682/) (2025)
- [XMAD-Bench: Cross-Domain Multilingual Audio Deepfake Benchmark](https://mlsanity.com/paper/2506.00462/) (2025)
- [Fake Speech Wild: Detecting Deepfake Speech on Social Media Platform](https://mlsanity.com/paper/2508.10559/) (2025)
- [QAMO: Quality-aware Multi-centroid One-class Learning For Speech Deepfake Detection](https://mlsanity.com/paper/2509.20679/) (2025)
- [XLSR-Kanformer: A KAN-Intergrated model for Synthetic Speech Detection](https://mlsanity.com/paper/2510.06706/) (2025)
- [SONAR: Spectral-Contrastive Audio Residuals for Generalizable Deepfake Detection](https://mlsanity.com/paper/2511.21325/) (2025)
- [Teffic-Audio: Tell Fact from Fiction](https://mlsanity.com/paper/2607.28351/) (2026)
- [Acoustic scattering AI for non-invasive object classifications: A case study on hair assessment](https://mlsanity.com/paper/2506.14148/) (2025)
- [Can Current Detectors Catch Face-to-Voice Deepfake Attacks?](https://mlsanity.com/paper/2510.21004/) (2025)
- [HyperPotter: Spell the Charm of High-Order Interactions in Audio Deepfake Detection](https://mlsanity.com/paper/2602.05670/) (2026)
- [ICLAD: In-Context Learning with Comparison-Guidance for Audio Deepfake Detection](https://mlsanity.com/paper/2604.16749/) (2026)
- [Time-Frequency Consistency Learning for Robust Speech Deepfake Detection](https://mlsanity.com/paper/2607.17761/) (2026)
- [Domain-Adaptive Dual-Gating Mixture of Experts for Generalizable Speech Deepfake Detection](https://mlsanity.com/paper/2609.33709/) (2026)
- [GLAD: Global-Local Adaptive Detector for Robust Speech Deepfake Detection](https://mlsanity.com/paper/2609.35411/) (2026)
- [Adaptive Compression and Targeted Perturbation: A Unified Framework for Generalized Audio Deepfake Detection](https://mlsanity.com/paper/2610.16049/) (2026)

## Abstract (arXiv)

Recent synthetic speech detectors leveraging the Transformer model have superior performance compared to the convolutional neural network counterparts. This improvement could be due to the powerful modeling ability of the multi-head self-attention (MHSA) in the Transformer model, which learns the temporal relationship of each input token. However, artifacts of synthetic speech can be located in specific regions of both frequency channels and temporal segments, while MHSA neglects this temporal-channel dependency of the input sequence. In this work, we proposed a Temporal-Channel Modeling (TCM) module to enhance MHSA's capability for capturing temporal-channel dependencies. Experimental results on the ASVspoof 2021 show that with only 0.03M additional parameters, the TCM module can outperform the state-of-the-art system by 9.25% in EER. Further ablation study reveals that utilizing both temporal and channel information yields the most improvement for detecting synthetic speech.