# PC2: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrieval

arXiv: https://arxiv.org/abs/2408.01349 · HTML: https://mlsanity.com/paper/2408.01349/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 61 references that are in the map.
- First posted on arXiv 2 Aug 2024.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval](https://mlsanity.com/paper/2003.03772/) (2020)
- [Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples](https://mlsanity.com/paper/2104.13963/) (2021)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Improving Image Captioning with Better Use of Captions](https://mlsanity.com/paper/2006.11807/) (2020)
- [SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation](https://mlsanity.com/paper/2209.15323/) (2022)
- [DMT: Dynamic Mutual Training for Semi-Supervised Learning](https://mlsanity.com/paper/2004.08514/) (2020)
- [Adversarial Representation Learning for Text-to-Image Matching](https://mlsanity.com/paper/1908.10534/) (2019)
- [A Comprehensive Survey on Cross-modal Retrieval](https://mlsanity.com/paper/1607.06215/) (2016)
- [A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation](https://mlsanity.com/paper/2310.16656/) (2023)

## Built on by (most cited first, 3 of 3)

- [ReCon: Enhancing True Correspondence Discrimination through Relation Consistency for Robust Noisy Correspondence Learning](https://mlsanity.com/paper/2502.19962/) (2025)
- [Ambiguity-Aware and High-Order Relation Learning for Multi-Grained Image-Text Matching](https://mlsanity.com/paper/2507.09256/) (2025)
- [PAUL: Uncertainty-Guided Partition and Augmentation for Robust Cross-View Geo-Localization under Noisy Correspondence](https://mlsanity.com/paper/2508.20066/) (2025)

## Abstract (arXiv)

In the realm of cross-modal retrieval, seamlessly integrating diverse modalities within multimedia remains a formidable challenge, especially given the complexities introduced by noisy correspondence learning (NCL). Such noise often stems from mismatched data pairs, which is a significant obstacle distinct from traditional noisy labels. This paper introduces Pseudo-Classification based Pseudo-Captioning (PC2) framework to address this challenge. PC2 offers a threefold strategy: firstly, it establishes an auxiliary "pseudo-classification" task that interprets captions as categorical labels, steering the model to learn image-text semantic similarity through a non-contrastive mechanism. Secondly, unlike prevailing margin-based techniques, capitalizing on PC2's pseudo-classification capability, we generate pseudo-captions to provide more informative and tangible supervision for each mismatched pair. Thirdly, the oscillation of pseudo-classification is borrowed to assistant the correction of correspondence. In addition to technical contributions, we develop a realistic NCL dataset called Noise of Web (NoW), which could be a new powerful NCL benchmark where noise exists naturally. Empirical evaluations of PC2 showcase marked improvements over existing state-of-the-art robust cross-modal retrieval techniques on both simulated and realistic datasets with various NCL settings. The contributed dataset and source code are released at https://github.com/alipay/PC2-NoiseofWeb.