# CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval

arXiv: https://arxiv.org/abs/2208.09843 · HTML: https://mlsanity.com/paper/2208.09843/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 66 references that are in the map.
- First posted on arXiv 21 Aug 2022.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2012.15409/) (2020)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [Visual Semantic Reasoning for Image-Text Matching](https://mlsanity.com/paper/1909.02701/) (2019)
- [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015)
- [Learning the Best Pooling Strategy for Visual Semantic Embedding](https://mlsanity.com/paper/2011.04305/) (2020)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval](https://mlsanity.com/paper/2003.03772/) (2020)
- [From Captions to Visual Concepts and Back](https://mlsanity.com/paper/1411.4952/) (2014)
- [Understanding and Evaluating Racial Biases in Image Captioning](https://mlsanity.com/paper/2106.08503/) (2021)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)](https://mlsanity.com/paper/1412.6632/) (2014)

## Built on by (most cited first, 13 of 13)

- [REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning](https://mlsanity.com/paper/2311.00519/) (2023)
- [USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval](https://mlsanity.com/paper/2301.06844/) (2023)
- [Composing Object Relations and Attributes for Image-Text Matching](https://mlsanity.com/paper/2406.11820/) (2024)
- [Plug-and-Play Regulators for Image-Text Matching](https://mlsanity.com/paper/2303.13371/) (2023)
- [RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models](https://mlsanity.com/paper/2304.10727/) (2023)
- [3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting](https://mlsanity.com/paper/2404.17273/) (2024)
- [SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Prior](https://mlsanity.com/paper/2510.15749/) (2025)
- [A Survey on Image-text Multimodal Models](https://mlsanity.com/paper/2309.15857/) (2023)
- [Deep Boosting Learning: A Brand-new Cooperative Approach for Image-Text Matching](https://mlsanity.com/paper/2404.18114/) (2024)
- [GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning](https://mlsanity.com/paper/2410.15266/) (2024)
- [GMM-Based Comprehensive Feature Extraction and Relative Distance Preservation For Few-Shot Cross-Modal Retrieval](https://mlsanity.com/paper/2505.13306/) (2025)
- [iEBAKER: Improved Remote Sensing Image-Text Retrieval Framework via Eliminate Before Align and Keyword Explicit Reasoning](https://mlsanity.com/paper/2504.05644/) (2025)
- [Open World Knowledge Aided Single-Cell Foundation Model with Robust Cross-Modal Cell-Language Pre-training](https://mlsanity.com/paper/2601.05648/) (2026)

## Abstract (arXiv)

Image-Text Retrieval (ITR) is challenging in bridging visual and lingual modalities. Contrastive learning has been adopted by most prior arts. Except for limited amount of negative image-text pairs, the capability of constrastive learning is restricted by manually weighting negative pairs as well as unawareness of external knowledge. In this paper, we propose our novel Coupled Diversity-Sensitive Momentum Constrastive Learning (CODER) for improving cross-modal representation. Firstly, a novel diversity-sensitive contrastive learning (DCL) architecture is invented. We introduce dynamic dictionaries for both modalities to enlarge the scale of image-text pairs, and diversity-sensitiveness is achieved by adaptive negative pair weighting. Furthermore, two branches are designed in CODER. One learns instance-level embeddings from image/text, and it also generates pseudo online clustering labels for its input image/text based on their embeddings. Meanwhile, the other branch learns to query from commonsense knowledge graph to form concept-level descriptors for both modalities. Afterwards, both branches leverage DCL to align the cross-modal embedding spaces while an extra pseudo clustering label prediction loss is utilized to promote concept-level representation learning for the second branch. Extensive experiments conducted on two popular benchmarks, i.e. MSCOCO and Flicker30K, validate CODER remarkably outperforms the state-of-the-art approaches. Our code is available at: https://github.com/BruceW91/CODER.