# Cross-Modal Retrieval with Cauchy-Schwarz Divergence

arXiv: https://arxiv.org/abs/2509.21339 · HTML: https://mlsanity.com/paper/2509.21339/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 87 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Generative Moment Matching Networks](https://mlsanity.com/paper/1502.02761/) (2015)
- [TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis](https://mlsanity.com/paper/2305.00976/) (2023)
- [Person Search with Natural Language Description](https://mlsanity.com/paper/1702.05729/) (2017)
- [Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval](https://mlsanity.com/paper/2303.12501/) (2023)
- [Reconsidering Representation Alignment for Multi-view Clustering](https://mlsanity.com/paper/2103.07738/) (2021)
- [Deep Cross-Modal Hashing](https://mlsanity.com/paper/1602.02255/) (2016)
- [Multimodal Recommender Systems: A Survey](https://mlsanity.com/paper/2302.03883/) (2023)
- [CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning](https://mlsanity.com/paper/1710.05106/) (2017)
- [Tri-Modal Motion Retrieval by Learning a Joint Embedding Space](https://mlsanity.com/paper/2403.00691/) (2024)
- [Unsupervised Generative Adversarial Cross-modal Hashing](https://mlsanity.com/paper/1712.00358/) (2017)
- [MHTN: Modal-adversarial Hybrid Transfer Network for Cross-modal Retrieval](https://mlsanity.com/paper/1708.04308/) (2017)
- [CCL: Cross-modal Correlation Learning with Multi-grained Fusion by Hierarchical Network](https://mlsanity.com/paper/1704.02116/) (2017)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [MMD-ReID: A Simple but Effective Solution for Visible-Thermal Person ReID](https://mlsanity.com/paper/2111.05059/) (2021)
- [Ev-3DOD: Pushing the Temporal Boundaries of 3D Object Detection with Event Cameras](https://mlsanity.com/paper/2502.19630/) (2025)
- [Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2502.17028/) (2025)
- [Integrating Information Theory and Adversarial Learning for Cross-modal Retrieval](https://mlsanity.com/paper/2104.04991/) (2021)
- [The Conditional Cauchy-Schwarz Divergence with Applications to Time-Series Data and Sequential Decision Making](https://mlsanity.com/paper/2301.08970/) (2023)
- [Domain Adaptation with Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2405.19978/) (2024)

## Abstract (arXiv)

Effective cross-modal retrieval requires robust alignment of heterogeneous data types. Most existing methods focus on bi-modal retrieval tasks and rely on distributional alignment techniques such as Kullback-Leibler divergence, Maximum Mean Discrepancy, and correlation alignment. However, these methods often suffer from critical limitations, including numerical instability, sensitivity to hyperparameters, and their inability to capture the full structure of the underlying distributions. In this paper, we introduce the Cauchy-Schwarz (CS) divergence, a hyperparameter-free measure that improves both training stability and retrieval performance. We further propose a novel Generalized CS (GCS) divergence inspired by H"older's inequality. This extension enables direct alignment of three or more modalities within a unified mathematical framework through a bidirectional circular comparison scheme, eliminating the need for exhaustive pairwise comparisons. Extensive experiments on six benchmark datasets demonstrate the effectiveness of our method in both bi-modal and tri-modal retrieval tasks. The code of our CS/GCS divergence is publicly available at https://github.com/JiahaoZhang666/CSD.