# CCL: Cross-modal Correlation Learning with Multi-grained Fusion by Hierarchical Network

arXiv: https://arxiv.org/abs/1704.02116 · HTML: https://mlsanity.com/paper/1704.02116/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2017.

## Built on by (most cited first, 10 of 10)

- [The Curious Layperson: Fine-Grained Image Recognition without Expert Labels](https://mlsanity.com/paper/2111.03651/) (2021)
- [A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval](https://mlsanity.com/paper/2201.02772/) (2022)
- [Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization](https://mlsanity.com/paper/2105.05631/) (2021)
- [Discriminative Semantic Transitive Consistency for Cross-Modal Learning](https://mlsanity.com/paper/2103.14103/) (2021)
- [Graph Pattern Loss based Diversified Attention Network for Cross-Modal Retrieval](https://mlsanity.com/paper/2106.13552/) (2021)
- [Deep Manifold Hashing: A Divide-and-Conquer Approach for Semi-Paired Unsupervised Cross-Modal Retrieval](https://mlsanity.com/paper/2209.12599/) (2022)
- [TAP: The Attention Patch for Cross-Modal Knowledge Transfer from Unlabeled Modality](https://mlsanity.com/paper/2302.02224/) (2023)
- [Tracing Human Stress from Physiological Signals using UWB Radar](https://mlsanity.com/paper/2410.10155/) (2024)
- [Multimodal Information Retrieval for Open World with Edit Distance Weak Supervision](https://mlsanity.com/paper/2506.20070/) (2025)
- [Cross-Modal Retrieval with Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2509.21339/) (2025)

## Abstract (arXiv)

Cross-modal retrieval has become a highlighted research topic for retrieval across multimedia data such as image and text. A two-stage learning framework is widely adopted by most existing methods based on Deep Neural Network (DNN): The first learning stage is to generate separate representation for each modality, and the second learning stage is to get the cross-modal common representation. However, the existing methods have three limitations: (1) In the first learning stage, they only model intra-modality correlation, but ignore inter-modality correlation with rich complementary context. (2) In the second learning stage, they only adopt shallow networks with single-loss regularization, but ignore the intrinsic relevance of intra-modality and inter-modality correlation. (3) Only original instances are considered while the complementary fine-grained clues provided by their patches are ignored. For addressing the above problems, this paper proposes a cross-modal correlation learning (CCL) approach with multi-grained fusion by hierarchical network, and the contributions are as follows: (1) In the first learning stage, CCL exploits multi-level association with joint optimization to preserve the complementary context from intra-modality and inter-modality correlation simultaneously. (2) In the second learning stage, a multi-task learning strategy is designed to adaptively balance the intra-modality semantic category constraints and inter-modality pairwise similarity constraints. (3) CCL adopts multi-grained modeling, which fuses the coarse-grained instances and fine-grained patches to make cross-modal correlation more precise. Comparing with 13 state-of-the-art methods on 6 widely-used cross-modal datasets, the experimental results show our CCL approach achieves the best performance.