# PhonemeFake: Redefining Deepfake Realism with Language-Driven Segmental Manipulation and Adaptive Bilevel Detection

arXiv: https://arxiv.org/abs/2506.22783 · HTML: https://mlsanity.com/paper/2506.22783/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 28 Jun 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks](https://mlsanity.com/paper/2110.01200/) (2021)
- [DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection](https://mlsanity.com/paper/2307.01426/) (2023)
- [Does Audio Deepfake Detection Generalize?](https://mlsanity.com/paper/2203.16263/) (2022)
- [ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild](https://mlsanity.com/paper/2210.02437/) (2022)
- [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](https://mlsanity.com/paper/2111.02813/) (2021)
- [Artificial Intelligence in the Creative Industries: A Review](https://mlsanity.com/paper/2007.12391/) (2020)
- [Half-Truth: A Partially Fake Audio Detection Dataset](https://mlsanity.com/paper/2104.03617/) (2021)
- [UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization](https://mlsanity.com/paper/2308.14395/) (2023)
- [Improved DeepFake Detection Using Whisper Features](https://mlsanity.com/paper/2306.01428/) (2023)
- [RawBMamba: End-to-End Bidirectional State Space Model for Audio Deepfake Detection](https://mlsanity.com/paper/2406.06086/) (2024)
- [SpoofCeleb: Speech Deepfake Detection and SASV In The Wild](https://mlsanity.com/paper/2409.17285/) (2024)
- [Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification](https://mlsanity.com/paper/2204.02639/) (2022)
- [Detection of Cross-Dataset Fake Audio Based on Prosodic and Pronunciation Features](https://mlsanity.com/paper/2305.13700/) (2023)
- [TexShape: Information Theoretic Sentence Embedding for Language Models](https://mlsanity.com/paper/2402.05132/) (2024)
- [Towards single integrated spoofing-aware speaker verification embeddings](https://mlsanity.com/paper/2305.19051/) (2023)
- [Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio](https://mlsanity.com/paper/2406.08112/) (2024)
- [SecureSpectra: Safeguarding Digital Identity from Deep Fake Threats via Intelligent Signatures](https://mlsanity.com/paper/2407.00913/) (2024)

## Built on by (most cited first, 3 of 3)

- [Towards Explicit Acoustic Evidence Perception in Audio LLMs for Speech Deepfake Detection](https://mlsanity.com/paper/2601.23066/) (2026)
- [PC-Mix: Partial-Component Audio Spoofing Detection under Mixed Speech and Environmental Sound Conditions](https://mlsanity.com/paper/2607.10345/) (2026)
- [ModalFidelity: Routing Modalities for Deepfake Detection on a Budget](https://mlsanity.com/paper/2609.38246/) (2026)

## Abstract (arXiv)

Deepfake (DF) attacks pose a growing threat as generative models become increasingly advanced. However, our study reveals that existing DF datasets fail to deceive human perception, unlike real DF attacks that influence public discourse. It highlights the need for more realistic DF attack vectors. We introduce PhonemeFake (PF), a DF attack that manipulates critical speech segments using language reasoning, significantly reducing human perception by up to 42% and benchmark accuracies by up to 94%. We release an easy-to-use PF dataset on HuggingFace and open-source bilevel DF segment detection model that adaptively prioritizes compute on manipulated regions. Our extensive experiments across three known DF datasets reveal that our detection model reduces EER by 91% while achieving up to 90% speed-up, with minimal compute overhead and precise localization beyond existing models as a scalable solution.