mlsanityData to 7 Oct 2026

Probabilistic Embeddings for Cross-Modal Retrieval

arXiv 2101.05068 ↗First posted 13 Jan 2021cs.CVcited by 150

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Cross-modal retrieval methods build a common representation space for samples from multiple modalities, typically from the vision and the language domains. For images and their captions, the multiplicity of the correspondences makes the task particularly challenging. Given an image (respectively a caption), there are multiple captions (respectively images) that equally make sense. In this paper, we argue that deterministic functions are not sufficiently powerful to capture such one-to-many correspondences. Instead, we propose to use Probabilistic Cross-Modal Embedding (PCME), where samples from the different modalities are represented as probabilistic distributions in the common embedding space. Since common benchmarks such as COCO suffer from non-exhaustive annotations for cross-modal matches, we propose to additionally evaluate retrieval on the CUB dataset, a smaller yet clean database where all possible image-caption pairs are annotated. We extensively ablate PCME and demonstrate that it not only improves the retrieval performance over its deterministic counterpart but also provides uncertainty estimates that render the embeddings more interpretable. Code is available at https://github.com/naver-ai/pcme

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Builds on 47 of its 63 references that are in the map. First posted on arXiv 13 Jan 2021.

Builds on

47 of 63 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2016Layer Normalizationcited by 5,579
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2013Auto-Encoding Variational Bayescited by 3,980
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  10. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
35 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  3. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  5. 2015Generating Sentences from a Continuous Spacecited by 607
  6. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  7. 2020A Metric Learning Reality Checkcited by 317
  8. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  9. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  10. 2018Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoderscited by 206
  11. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  12. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  13. 2019Probabilistic Face Embeddingscited by 171
  14. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  15. 2020Data Uncertainty Learning in Face Recognitioncited by 134
  16. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 113
  17. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  18. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  19. 2014Word Representations via Gaussian Embeddingcited by 96
  20. 2019Polysemous Visual-Semantic Embedding for Cross-Modal Retrievalcited by 83
  21. 2018NSML: Meet the MLaaS platform with a real-world case studycited by 70
  22. 2018Modeling Uncertainty with Hedged Instance Embeddingcited by 52
  23. 2019Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matchingcited by 49
  24. 2019View-Invariant Probabilistic Embedding for Human Posecited by 47
  25. 2016Instance-aware Image and Sentence Matching with Selective Multimodal LSTMcited by 43
  26. 2015Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Spacecited by 35
  27. 2018Finding beans in burgers: Deep semantic-visual embedding with localizationcited by 28
  28. 2020Adaptive Offline Quintuplet Loss for Image-Text Matchingcited by 26
  29. 2020Preserving Semantic Neighborhoods for Robust Cross-modal Retrievalcited by 20
  30. 2014Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotationcited by 16
  31. 2017A Mixture Model for Learning Multi-Sense Word Embeddingscited by 12
  32. 2020Probabilistic embeddings for speaker diarizationcited by 11
  33. 2019Stochastic Prototype Embeddingscited by 10
  34. 2017Separating Self-Expression and Visual Content in Hashtag Supervisioncited by 9
  35. 2020AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weightscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 29; 2023: 29; 2024: 33; 2025: 37; 2026: 16.

150 papers.

Sort:

By area:

  1. 2023Probabilistic Concept Bottleneck Modelscited by 101
  2. 2023CompoDiff: Versatile Composed Image Retrieval With Latent Diffusioncited by 84
  3. 2022ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similaritycited by 82
  4. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  5. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  6. 2022Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representationscited by 57
  7. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  8. 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55
  9. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  10. 2021On Semantic Similarity in Video Retrievalcited by 45
  11. 2022Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularizationcited by 45
  12. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Modelcited by 41
38 more
  1. 2021Learning Fair Classifiers with Partially Annotated Group Labelscited by 40
  2. 2023Improved Probabilistic Image-Text Representationscited by 40
  3. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  4. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  5. 2022Probabilistic Representations for Video Contrastive Learningcited by 33
  6. 2022Improving Cross-Modal Retrieval with Set of Diverse Embeddingscited by 33
  7. 2025HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrievalcited by 32
  8. 2023iKUN: Speak to Trackers without Retrainingcited by 31
  9. 2022SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Textcited by 30
  10. 2022ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrievalcited by 29
  11. 2022Probabilistic Compositional Embeddings for Multimodal Image Retrievalcited by 29
  12. 2023Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputscited by 28
  13. 2023Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentationcited by 27
  14. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  15. 2023Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrievalcited by 22
  16. 2024Image-Text Co-Decomposition for Text-Supervised Semantic Segmentationcited by 21
  17. 2022Text-Adaptive Multiple Visual Prototype Matching for Video-Text Retrievalcited by 20
  18. 2022A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learningcited by 19
  19. 2023Vision-Language Dataset Distillationcited by 18
  20. 2021Learning Structural Representations for Recipe Generation and Food Retrievalcited by 17
  21. 2023Pluralistic Aging Diffusion Autoencodercited by 17
  22. 2023Probabilistic Prompt Learning for Dense Predictioncited by 17
  23. 2023URL: A Representation Learning Benchmark for Transferable Uncertainty Estimatescited by 17
  24. 2024Robust Multimodal Learning via Representation Decouplingcited by 17
  25. 2024Probabilistic Language-Image Pre-Trainingcited by 16
  26. 2022Out of Distribution Data Detection Using Dropout Bayesian Neural Networkscited by 15
  27. 2023Dynamic Weighted Combiner for Mixed-Modal Image Retrievalcited by 15
  28. 2021Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matchingcited by 13
  29. 2024Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrievalcited by 13
  30. 2024RICA2: Rubric-Informed, Calibrated Assessment of Actionscited by 13
  31. 2024Post-hoc Probabilistic Vision-Language Modelscited by 13
  32. 2024DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animationcited by 12
  33. 2024Improving Visual Recognition with Hyperbolical Visual Hierarchy Mappingcited by 11
  34. 2024Composing Object Relations and Attributes for Image-Text Matchingcited by 11
  35. 2021Continual learning in cross-modal retrievalcited by 10
  36. 2022Probabilistic Integration of Object Level Annotations in Chest X-ray Classificationcited by 10
  37. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
  38. 2022Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slidescited by 9

50 most cited of 150 All 150 (JSON)