# LIMITR: Leveraging Local Information for Medical Image-Text Representation

arXiv: https://arxiv.org/abs/2303.11755 · HTML: https://mlsanity.com/paper/2303.11755/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 21 Mar 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [Visual Semantic Reasoning for Image-Text Matching](https://mlsanity.com/paper/1909.02701/) (2019)
- [A Survey of Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2202.10936/) (2022)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [Joint Learning of Localized Representations from Medical Images and Reports](https://mlsanity.com/paper/2112.02889/) (2021)
- [Learning Semantic Concepts and Order for Image and Sentence Matching](https://mlsanity.com/paper/1712.02036/) (2017)

## Built on by (most cited first, 13 of 13)

- [Improving fine-grained understanding in image-text pre-training](https://mlsanity.com/paper/2401.09865/) (2024)
- [Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation](https://mlsanity.com/paper/2404.02845/) (2024)
- [Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation](https://mlsanity.com/paper/2312.08078/) (2023)
- [Medical Vision Language Pretraining: A survey](https://mlsanity.com/paper/2312.06224/) (2023)
- [HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling](https://mlsanity.com/paper/2403.13319/) (2024)
- [Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training](https://mlsanity.com/paper/2405.19654/) (2024)
- [MedRAT: Unpaired Medical Report Generation via Auxiliary Tasks](https://mlsanity.com/paper/2407.03919/) (2024)
- [Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models](https://mlsanity.com/paper/2404.12920/) (2024)
- [Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrieval](https://mlsanity.com/paper/2312.15840/) (2023)
- [Image-aware Evaluation of Generated Medical Reports](https://mlsanity.com/paper/2410.17357/) (2024)
- [MedCycle: Unpaired Medical Report Generation via Cycle-Consistency](https://mlsanity.com/paper/2403.13444/) (2024)
- [Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering](https://mlsanity.com/paper/2510.08791/) (2025)
- [Anatomy-Aware Conditional Image-Text Retrieval](https://mlsanity.com/paper/2503.07456/) (2025)

## Abstract (arXiv)

Medical imaging analysis plays a critical role in the diagnosis and treatment of various medical conditions. This paper focuses on chest X-ray images and their corresponding radiological reports. It presents a new model that learns a joint X-ray image & report representation. The model is based on a novel alignment scheme between the visual data and the text, which takes into account both local and global information. Furthermore, the model integrates domain-specific information of two types -- lateral images and the consistent visual structure of chest images. Our representation is shown to benefit three types of retrieval tasks: text-image retrieval, class-based retrieval, and phrase-grounding.