mlsanityData to 7 Oct 2026

Multimodal Representation Learning via Maximization of Local Mutual Information

arXiv 2103.04537 ↗First posted 8 Mar 2021eess.IV cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

21 of 34 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  8. 2019Publicly Available Clinical BERT Embeddingscited by 995
  9. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  10. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  11. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  12. 2018Isolating Sources of Disentanglement in Variational Autoencoderscited by 167
9 more
  1. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  2. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  3. 2017Unsupervised Learning by Predicting Noisecited by 97
  4. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  5. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  6. 2020Deep Learning to Quantify Pulmonary Edema in Chest Radiographscited by 10
  7. 2020DEMI: Discriminative Estimator of Mutual Informationcited by 5
  8. 2018Bimodal network architectures for automatic generation of image annotation from textcited by 4
  9. 2019Semi-supervised Learning for Quantification of Pulmonary Edema in Chest X-Ray Imagescited by 2

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 7; 2023: 11; 2024: 3; 2025: 4.

27 papers.

Sort:

By area:

  1. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  2. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  3. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  4. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  5. 2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
  6. 2023CLIP in Medical Imaging: A Surveycited by 40
  7. 2023Multimodal Variational Auto-encoder based Audio-Visual Segmentationcited by 37
  8. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  9. 2023Medical Vision Language Pretraining: A surveycited by 24
  10. 2024ChEX: Interactive Localization and Region Description in Chest X-rayscited by 18
  11. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  12. 2023Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regimecited by 15
15 more
  1. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  2. 2022Using Multiple Instance Learning to Build Multimodal Representationscited by 7
  3. 2022FlexR: Few-shot Classification with Language Embeddings for Structured Reporting of Chest X-rayscited by 5
  4. 2022Context-sensitive neocortical neurons transform the effectiveness and efficiency of neural information processingcited by 5
  5. 2022The Role of Local Alignment and Uniformity in Image-Text Contrastive Learning on Medical Imagescited by 5
  6. 2025DADM: Dual Alignment of Domain and Modality for Face Anti-spoofingcited by 4
  7. 2023Improving Face Recognition from Caption Supervision with Multi-Granular Contextual Feature Aggregationcited by 2
  8. 2023Sample-Specific Debiasing for Better Image-Text Modelscited by 1
  9. 2024MOSMOS: Multi-organ segmentation facilitated by medical report supervisioncited by 1
  10. 2025Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answeringcited by 1
  11. 2022RadTex: Learning Efficient Radiograph Representations from Text Reportscited by 0
  12. 2022That's the Wrong Lung! Evaluating and Improving the Interpretability of Unsupervised Multimodal Encoders for Medical Datacited by 0
  13. 2023Improving Medical Visual Representations via Radiology Report Generationcited by 0
  14. 2025Anatomy-Aware Conditional Image-Text Retrievalcited by 0
  15. 2025AGA: An adaptive group alignment framework for structured medical cross-modal representation learningcited by 0