# The DeepSpeak Dataset arXiv: https://arxiv.org/abs/2408.05366 ยท HTML: https://mlsanity.com/paper/2408.05366/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 43 of its 68 references that are in the map. - First posted on arXiv 9 Aug 2024. ## Builds on (most cited first, 30 of 43) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022) - [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019) - [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018) - [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017) - [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022) - [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020) - [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021) - [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019) - [Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics](https://mlsanity.com/paper/1909.12962/) (2019) - [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018) - [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018) - [DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection](https://mlsanity.com/paper/2001.03024/) (2020) - [CelebV-HQ: A Large-Scale Video Facial Attributes Dataset](https://mlsanity.com/paper/2207.12393/) (2022) - [SimSwap: An Efficient Framework For High Fidelity Face Swapping](https://mlsanity.com/paper/2106.06340/) (2021) - [Towards Robust Blind Face Restoration with Codebook Lookup Transformer](https://mlsanity.com/paper/2206.11253/) (2022) - [WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection](https://mlsanity.com/paper/2101.01456/) (2021) - [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://mlsanity.com/paper/2407.03168/) (2024) - [FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset](https://mlsanity.com/paper/2108.05080/) (2021) - [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021) - [VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolution](https://mlsanity.com/paper/2205.03409/) (2022) - [End-to-end anti-spoofing with RawNet2](https://mlsanity.com/paper/2011.01108/) (2020) - [DF40: Toward Next-Generation Deepfake Detection](https://mlsanity.com/paper/2406.13495/) (2024) - [ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis](https://mlsanity.com/paper/2103.05630/) (2021) - [ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild](https://mlsanity.com/paper/2210.02437/) (2022) - [FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces](https://mlsanity.com/paper/1803.09179/) (2018) - [VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild](https://mlsanity.com/paper/2211.14758/) (2022) - [Towards Automatic Face-to-Face Translation](https://mlsanity.com/paper/2003.00418/) (2020) - [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](https://mlsanity.com/paper/2111.02813/) (2021) - [End-to-End Spectro-Temporal Graph Attention Networks for Speaker Verification Anti-Spoofing and Speech Deepfake Detection](https://mlsanity.com/paper/2107.12710/) (2021) ## Built on by (most cited first, 6 of 6) - [Beyond Face Swapping: A Diffusion-Based Digital Human Benchmark for Multimodal Deepfake Detection](https://mlsanity.com/paper/2505.16512/) (2025) - [Omni-Fake: Benchmarking Unified Multimodal Social Media Deepfake Detection](https://mlsanity.com/paper/2605.01638/) (2026) - [Scalable, Energy-Efficient Optical-Neural Architecture for Multiplexed Deepfake Video Detection](https://mlsanity.com/paper/2605.19360/) (2026) - [EMO-BOOST: Emotion-Augmented Audio-Visual Features for Improved Generalization in Deepfake Detection](https://mlsanity.com/paper/2605.19630/) (2026) - [The DeepSpeak-Agentic Dataset](https://mlsanity.com/paper/2606.03686/) (2026) - [CUPID: Reconstructing UV Texture Maps for Interpretable Person-of-Interest Deepfake Detection](https://mlsanity.com/paper/2606.20302/) (2026) ## Abstract (arXiv) Deepfakes represent a growing concern across domains such as disinformation, fraud, and non-consensual media. In particular, the rise of video conference and identity-driven attacks in high-stakes scenarios--such as impostor hiring--demands new forensic resources. Despite significant efforts to develop robust detection classifiers to distinguish the real from the fake, commonly used training datasets remain inadequate: relying on low-quality and outdated deepfake generators, consisting of content scraped from online repositories without participant consent, lacking in multimodal coverage, and rarely employing identity-matching protocols to ensure realistic fakes. To overcome these limitations, we present the DeepSpeak dataset, a diverse and multimodal dataset comprising over 100 hours of authentic and deepfake audiovisual content, specifically focused on the challenging and diverse talking heads context. We contribute: i) more than 50 hours of real, self-recorded data collected from 500 diverse and consenting participants, ii) more than 50 hours of state-of-the-art audio and visual deepfakes generated using 14 video synthesis engines and three voice cloning engines, and iii) an embedding-based, identity-matching approach to ensure the creation of convincing, high-quality identity face swaps that realistically simulate adversarial deepfake attacks. We also perform large-scale evaluations of state-of-the-art deepfake detectors and show that, without retraining, these detectors fail to generalize to this DeepSpeak dataset, highlighting the importance of a large and diverse dataset containing deepfakes from the latest generative-AI tools.