# Visual and audio scene classification for detecting discrepancies in video: a baseline method and experimental protocol

arXiv: https://arxiv.org/abs/2405.00384 · HTML: https://mlsanity.com/paper/2405.00384/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 1 May 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysis](https://mlsanity.com/paper/2011.00030/) (2020)
- [Unsupervised Adversarial Domain Adaptation Based On The Wasserstein Distance For Acoustic Scene Classification](https://mlsanity.com/paper/1904.10678/) (2019)
- [Unsupervised Domain Adaptation for Acoustic Scene Classification Using Band-Wise Statistics Matching](https://mlsanity.com/paper/2005.00145/) (2020)

## Abstract (arXiv)

This paper presents a baseline approach and an experimental protocol for a specific content verification problem: detecting discrepancies between the audio and video modalities in multimedia content. We first design and optimize an audio-visual scene classifier, to compare with existing classification baselines that use both modalities. Then, by applying this classifier separately to the audio and the visual modality, we can detect scene-class inconsistencies between them. To facilitate further research and provide a common evaluation platform, we introduce an experimental protocol and a benchmark dataset simulating such inconsistencies. Our approach achieves state-of-the-art results in scene classification and promising outcomes in audio-visual discrepancies detection, highlighting its potential in content verification applications.