# Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning

arXiv: https://arxiv.org/abs/2405.16996 · HTML: https://mlsanity.com/paper/2405.16996/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [On the Adversarial Robustness of Multi-Modal Foundation Models](https://mlsanity.com/paper/2308.10741/) (2023)
- [CyCLIP: Cyclic Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2205.14459/) (2022)
- [Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)](https://mlsanity.com/paper/2203.12221/) (2022)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning](https://mlsanity.com/paper/2303.05952/) (2023)
- [Deep Learning from Noisy Image Labels with Quality Embedding](https://mlsanity.com/paper/1711.00583/) (2017)
- [Robust Contrastive Learning against Noisy Views](https://mlsanity.com/paper/2201.04309/) (2022)
- [Improving Zero-shot Generalization and Robustness of Multi-modal Models](https://mlsanity.com/paper/2212.01758/) (2022)
- [Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval](https://mlsanity.com/paper/2304.03391/) (2023)
- [COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval](https://mlsanity.com/paper/2204.07441/) (2022)
- [BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency](https://mlsanity.com/paper/2303.12419/) (2023)
- [Contrastive Learning with Boosted Memorization](https://mlsanity.com/paper/2205.12693/) (2022)
- [Noisy Correspondence Learning with Meta Similarity Correction](https://mlsanity.com/paper/2304.06275/) (2023)
- [Combating Representation Learning Disparity with Geometric Harmonization](https://mlsanity.com/paper/2310.17622/) (2023)
- [Latent Class-Conditional Noise Model](https://mlsanity.com/paper/2302.09595/) (2023)

## Built on by (most cited first, 10 of 10)

- [ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.20358/) (2026)
- [Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring](https://mlsanity.com/paper/2506.08429/) (2025)
- [Disentangled Noisy Correspondence Learning](https://mlsanity.com/paper/2408.05503/) (2024)
- [ReCon: Enhancing True Correspondence Discrimination through Relation Consistency for Robust Noisy Correspondence Learning](https://mlsanity.com/paper/2502.19962/) (2025)
- [Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning](https://mlsanity.com/paper/2507.12998/) (2025)
- [Multi-Modal Dataset Distillation in the Wild](https://mlsanity.com/paper/2506.01586/) (2025)
- [Robust Noisy Correspondence Learning via Self-Drop and Dual-Weight](https://mlsanity.com/paper/2412.06172/) (2024)
- [PAUL: Uncertainty-Guided Partition and Augmentation for Robust Cross-View Geo-Localization under Noisy Correspondence](https://mlsanity.com/paper/2508.20066/) (2025)
- [Rethinking Time Series Domain Generalization via Structure-Stratified Calibration](https://mlsanity.com/paper/2603.02756/) (2026)
- [Intra-Modal Neighbors Never Lie: Rectifying Inter-Modal Noisy Correspondence via Graph-Based Intra-Modal Reasoning](https://mlsanity.com/paper/2606.04061/) (2026)

## Abstract (arXiv)

Noisy correspondence that refers to mismatches in cross-modal data pairs, is prevalent on human-annotated or web-crawled datasets. Prior approaches to leverage such data mainly consider the application of uni-modal noisy label learning without amending the impact on both cross-modal and intra-modal geometrical structures in multimodal learning. Actually, we find that both structures are effective to discriminate noisy correspondence through structural differences when being well-established. Inspired by this observation, we introduce a Geometrical Structure Consistency (GSC) method to infer the true correspondence. Specifically, GSC ensures the preservation of geometrical structures within and between modalities, allowing for the accurate discrimination of noisy samples based on structural differences. Utilizing these inferred true correspondence labels, GSC refines the learning of geometrical structures by filtering out the noisy samples. Experiments across four cross-modal datasets confirm that GSC effectively identifies noisy samples and significantly outperforms the current leading methods.