# 3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition

arXiv: https://arxiv.org/abs/1706.05739 · HTML: https://mlsanity.com/paper/1706.05739/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2017.

## Built on by (most cited first, 9 of 9)

- [RealImpact: A Dataset of Impact Sound Fields for Real Objects](https://mlsanity.com/paper/2306.09944/) (2023)
- [Personalized One-Shot Lipreading for an ALS Patient](https://mlsanity.com/paper/2111.01740/) (2021)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Spatio-Temporal Attention Mechanism and Knowledge Distillation for Lip Reading](https://mlsanity.com/paper/2108.03543/) (2021)
- [Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models](https://mlsanity.com/paper/2207.05692/) (2022)
- [Development and evaluation of a deep learning algorithm for German word recognition from lip movements](https://mlsanity.com/paper/2504.15792/) (2025)
- [Learning Audio-Visual Embeddings with Inferred Latent Interaction Graphs](https://mlsanity.com/paper/2601.11995/) (2026)
- [Improving Word Recognition in Speech Transcriptions by Decision-level Fusion of Stemming and Two-way Phoneme Pruning](https://mlsanity.com/paper/2107.12428/) (2021)
- [Machine Learning Computer Vision Applications for Spatial AI Object Recognition in Orange County, California](https://mlsanity.com/paper/2303.07560/) (2023)

## Abstract (arXiv)

Audio-visual recognition (AVR) has been considered as a solution for speech recognition tasks when the audio is corrupted, as well as a visual recognition method used for speaker verification in multi-speaker scenarios. The approach of AVR systems is to leverage the extracted information from one modality to improve the recognition ability of the other modality by complementing the missing information. The essential problem is to find the correspondence between the audio and visual streams, which is the goal of this work. We propose the use of a coupled 3D Convolutional Neural Network (3D-CNN) architecture that can map both modalities into a representation space to evaluate the correspondence of audio-visual streams using the learned multimodal features. The proposed architecture will incorporate both spatial and temporal information jointly to effectively find the correlation between temporal information for different modalities. By using a relatively small network architecture and much smaller dataset for training, our proposed method surpasses the performance of the existing similar methods for audio-visual matching which use 3D CNNs for feature representation. We also demonstrate that an effective pair selection method can significantly increase the performance. The proposed method achieves relative improvements over 20% on the Equal Error Rate (EER) and over 7% on the Average Precision (AP) in comparison to the state-of-the-art method.