# Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling

arXiv: https://arxiv.org/abs/2508.02000 · HTML: https://mlsanity.com/paper/2508.02000/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [TriDet: Temporal Action Detection with Relative Boundary Modeling](https://mlsanity.com/paper/2303.07347/) (2023)
- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation](https://mlsanity.com/paper/2103.06695/) (2021)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [ADD 2022: the First Audio Deep Synthesis Detection Challenge](https://mlsanity.com/paper/2202.08433/) (2022)
- [ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scale](https://mlsanity.com/paper/2408.08739/) (2024)
- [Combining EfficientNet and Vision Transformers for Video Deepfake Detection](https://mlsanity.com/paper/2107.02612/) (2021)
- [AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset](https://mlsanity.com/paper/2311.15308/) (2023)
- [BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation](https://mlsanity.com/paper/2009.07641/) (2020)
- [ADD 2023: the Second Audio Deepfake Detection Challenge](https://mlsanity.com/paper/2305.13774/) (2023)
- [ASVspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech](https://mlsanity.com/paper/2102.05889/) (2021)
- [The PartialSpoof Database and Countermeasures for the Detection of Short Fake Speech Segments Embedded in an Utterance](https://mlsanity.com/paper/2204.05177/) (2022)
- [UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization](https://mlsanity.com/paper/2308.14395/) (2023)
- [How Do Neural Spoofing Countermeasures Detect Partially Spoofed Audio?](https://mlsanity.com/paper/2406.02483/) (2024)
- [Manipulated Regions Localization For Partially Deepfake Audio: A Survey](https://mlsanity.com/paper/2506.14396/) (2025)
- [Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection](https://mlsanity.com/paper/2210.00753/) (2022)
- [Multimodal Transformer Distillation for Audio-Visual Synchronization](https://mlsanity.com/paper/2210.15563/) (2022)
- [A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2412.00049/) (2024)

## Built on by (most cited first, 2 of 2)

- [Deepfake Word Detection by Next-token Prediction using Fine-tuned Whisper](https://mlsanity.com/paper/2602.22658/) (2026)
- [Word-Anchored Temporal Forgery Localization](https://mlsanity.com/paper/2603.06220/) (2026)

## Abstract (arXiv)

Audio-visual temporal deepfake localization under the content-driven partial manipulation remains a highly challenging task. In this scenario, the deepfake regions are usually only spanning a few frames, with the majority of the rest remaining identical to the original. To tackle this, we propose a Hierarchical Boundary Modeling Network (HBMNet), which includes three modules: an Audio-Visual Feature Encoder that extracts discriminative frame-level representations, a Coarse Proposal Generator that predicts candidate boundary regions, and a Fine-grained Probabilities Generator that refines these proposals using bidirectional boundary-content probabilities. From the modality perspective, we enhance audio-visual learning through dedicated encoding and fusion, reinforced by frame-level supervision to boost discriminability. From the temporal perspective, HBMNet integrates multi-scale cues and bidirectional boundary-content relationships. Experiments show that encoding and fusion primarily improve precision, while frame-level supervision boosts recall. Each module (audio-visual fusion, temporal scales, bi-directionality) contributes complementary benefits, collectively enhancing localization performance. HBMNet outperforms BA-TFD and UMMAFormer and shows improved potential scalability with more training data.