mlsanityData to 14 Oct 2026

On the Behavior of Audio-Visual Fusion Architectures in Identity Verification Tasks

arXiv 2311.05071 ↗First posted 9 Nov 2023cs.LG cs.CV cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

We train an identity verification architecture and evaluate modifications to the part of the model that combines audio and visual representations, including in scenarios where one input is missing in either of two examples to be compared. We report results on the Voxceleb1-E test set that suggest averaging the output embeddings improves error rate in the full-modality setting and when a single modality is missing, and makes more complete use of the embedding space than systems which use shared layers and discuss possible reasons for this behavior.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

21 of 36 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  6. 2017Decoupled Weight Decay Regularizationcited by 1,831
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  8. 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 903
  9. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  10. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  11. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  12. 2018Deep Audio-Visual Speech Recognitioncited by 322
9 more
  1. 2017Generalized End-to-End Loss for Speaker Verificationcited by 293
  2. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  3. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 95
  4. 2017Speaker Diarization with LSTMcited by 73
  5. 2016A Comprehensive Analysis of Deep Learning Based Representation for Face Recognitioncited by 23
  6. 2021A Multi-View Approach To Audio-Visual Speaker Verificationcited by 23
  7. 2019Who said that?: Audio-visual speaker diarisation of real-world meetingscited by 22
  8. 2018Noise-tolerant Audio-visual Online Person Verification using an Attention-based Neural Network Fusioncited by 21
  9. 2017End-to-End Audiovisual Fusion with LSTMscited by 5

Built on by

Not cited yet.