# Deepfake Audio Detection Using Spectrogram-based Feature and Ensemble of Deep Learning Models

arXiv: https://arxiv.org/abs/2407.01777 · HTML: https://mlsanity.com/paper/2407.01777/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 10 of 10)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [ADD 2022: the First Audio Deep Synthesis Detection Challenge](https://mlsanity.com/paper/2202.08433/) (2022)
- [Audio Deepfake Detection: A Survey](https://mlsanity.com/paper/2308.14970/) (2023)
- [EmoFake: An Initial Dataset for Emotion Fake Audio Detection](https://mlsanity.com/paper/2211.05363/) (2022)
- [Wider or Deeper Neural Network Architecture for Acoustic Scene Classification with Mismatched Recording Devices](https://mlsanity.com/paper/2203.12314/) (2022)
- [SceneFake: An Initial Dataset and Benchmarks for Scene Fake Audio Detection](https://mlsanity.com/paper/2211.06073/) (2022)

## Built on by (most cited first, 5 of 5)

- [Towards Explicit Acoustic Evidence Perception in Audio LLMs for Speech Deepfake Detection](https://mlsanity.com/paper/2601.23066/) (2026)
- [Reliable Audio Deepfake Detection in Variable Conditions via Quantum-Kernel SVMs](https://mlsanity.com/paper/2512.18797/) (2025)
- [Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Study](https://mlsanity.com/paper/2503.22713/) (2025)
- [Melody or Machine: Detecting Synthetic Music with Dual-Stream Contrastive Learning](https://mlsanity.com/paper/2512.00621/) (2025)
- [Enhancing Efficiency and Performance in Deepfake Audio Detection through Neuron-level Dropin & Neuroplasticity Mechanisms](https://mlsanity.com/paper/2603.24343/) (2026)

## Abstract (arXiv)

In this paper, we propose a deep learning based system for the task of deepfake audio detection. In particular, the draw input audio is first transformed into various spectrograms using three transformation methods of Short-time Fourier Transform (STFT), Constant-Q Transform (CQT), Wavelet Transform (WT) combined with different auditory-based filters of Mel, Gammatone, linear filters (LF), and discrete cosine transform (DCT). Given the spectrograms, we evaluate a wide range of classification models based on three deep learning approaches. The first approach is to train directly the spectrograms using our proposed baseline models of CNN-based model (CNN-baseline), RNN-based model (RNN-baseline), C-RNN model (C-RNN baseline). Meanwhile, the second approach is transfer learning from computer vision models such as ResNet-18, MobileNet-V3, EfficientNet-B0, DenseNet-121, SuffleNet-V2, Swint, Convnext-Tiny, GoogLeNet, MNASsnet, RegNet. In the third approach, we leverage the state-of-the-art audio pre-trained models of Whisper, Seamless, Speechbrain, and Pyannote to extract audio embeddings from the input spectrograms. Then, the audio embeddings are explored by a Multilayer perceptron (MLP) model to detect the fake or real audio samples. Finally, high-performance deep learning models from these approaches are fused to achieve the best performance. We evaluated our proposed models on ASVspoof 2019 benchmark dataset. Our best ensemble model achieved an Equal Error Rate (EER) of 0.03, which is highly competitive to top-performing systems in the ASVspoofing 2019 challenge. Experimental results also highlight the potential of selective spectrograms and deep learning approaches to enhance the task of audio deepfake detection.