mlsanityData to 7 Oct 2026

Learnable PINs: Cross-Modal Embeddings for Person Identity

arXiv 1805.00833 ↗First posted 2 May 2018cs.CVcited by 60

Not on the map. Why?

Abstract (arXiv)

We propose and investigate an identity sensitive joint embedding of face and voice. Such an embedding enables cross-modal retrieval from voice to face and from face to voice. We make the following four contributions: first, we show that the embedding can be learnt from videos of talking faces, without requiring any identity labels, using a form of cross-modal self-supervision; second, we develop a curriculum learning schedule for hard negative mining targeted to this task, that is essential for learning to proceed successfully; third, we demonstrate and evaluate cross-modal retrieval for identities unseen and unheard during training over a number of scenarios and establish a benchmark for this novel task; finally, we show an application of using the joint embedding for automatically retrieving and labelling characters in TV dramas.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 10; 2023: 8; 2024: 9; 2025: 6; 2026: 5.

60 papers.

Sort:

By area:

  1. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  2. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  3. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  4. 2022Audio Self-supervised Learning: A Surveycited by 65
  5. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  6. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  7. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  8. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  9. 2021Audio-Visual Synchronisation in the wildcited by 39
  10. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 35
  11. 2021Sound-Guided Semantic Image Manipulationcited by 35
  12. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
38 more
  1. 2021Fusion and Orthogonal Projection for Improved Face-Voice Associationcited by 29
  2. 2022Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Datacited by 27
  3. 2023Imaginary Voice: Face-styled Diffusion Model for Text-to-Speechcited by 27
  4. 2021Multimodal Knowledge Expansioncited by 26
  5. 2021Face, Body, Voice: Video Person-Clustering with Multiple Modalitiescited by 24
  6. 2021AVA-AVD: Audio-Visual Speaker Diarization in the Wildcited by 24
  7. 2022Is an Object-Centric Video Representation Beneficial for Transfer?cited by 24
  8. 2021A Multi-View Approach To Audio-Visual Speaker Verificationcited by 23
  9. 2021Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Associationcited by 21
  10. 2023Single-branch Network for Multimodal Trainingcited by 13
  11. 2024A Multimodal Single-Branch Embedding Network for Recommendation in Cold-Start and Missing Modality Scenarioscited by 12
  12. 2021ASMR: Learning Attribute-Based Person Search with Adaptive Semantic Margin Regularizercited by 11
  13. 2022A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!cited by 9
  14. 2023Speaker Recognition in Realistic Scenario Using Multimodal Datacited by 9
  15. 2025PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Associationcited by 9
  16. 2022Robust Sound-Guided Image Manipulationcited by 7
  17. 2021TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separationcited by 6
  18. 2022Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrastcited by 6
  19. 2025Face-voice Association in Multilingual Environments (FAME) 2026 Challenge Evaluation Plancited by 6
  20. 2025Linking Faces and Voices Across Languages: Insights from the FAME 2026 Challengecited by 5
  21. 2021Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representationscited by 4
  22. 2021FaVoA: Face-Voice Association Favours Ambiguous Speaker Detectioncited by 4
  23. 2023Audio-Visual Speaker Verification via Joint Cross-Attentioncited by 4
  24. 2024Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plancited by 4
  25. 2024Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approachcited by 4
  26. 2021APES: Audiovisual Person Search in Untrimmed Videocited by 3
  27. 2022Learning Branched Fusion and Orthogonal Projection for Face-Voice Associationcited by 3
  28. 2022A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognitioncited by 3
  29. 2023Audio Representation Learning by Distilling Video as Privileged Informationcited by 3
  30. 2024Audio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attentioncited by 3
  31. 2024Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalitiescited by 2
  32. 2021Discriminative Semantic Transitive Consistency for Cross-Modal Learningcited by 1
  33. 2024"Previously on ..." From Recaps to Story Summarizationcited by 1
  34. 2024Exploring Robust Face-Voice Matching in Multilingual Environmentscited by 1
  35. 2024Contrastive Learning-based Chaining-Cluster for Multilingual Voice-Face Associationcited by 1
  36. 2025RFOP: Rethinking Fusion and Orthogonal Projection for Face-Voice Associationcited by 1
  37. 2026Face-Voice Association with Inductive Bias for Maximum Class Separationcited by 1
  38. 2026Evaluating Multimodal Narrative Understanding of Popular Hollywood Filmscited by 1

50 most cited of 60 All 60 (JSON)