# Simple to Complex Cross-modal Learning to Rank

arXiv: https://arxiv.org/abs/1702.01229 · HTML: https://mlsanity.com/paper/1702.01229/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Feb 2017.

## Built on by (most cited first, 3 of 3)

- [Discriminative Semantic Transitive Consistency for Cross-Modal Learning](https://mlsanity.com/paper/2103.14103/) (2021)
- [SSR-HEF: Crowd Counting with Multi-Scale Semantic Refining and Hard Example Focusing](https://mlsanity.com/paper/2204.07406/) (2022)
- [Reliable Shot Identification for Complex Event Detection via Visual-Semantic Embedding](https://mlsanity.com/paper/2110.08063/) (2021)

## Abstract (arXiv)

The heterogeneity-gap between different modalities brings a significant challenge to multimedia information retrieval. Some studies formalize the cross-modal retrieval tasks as a ranking problem and learn a shared multi-modal embedding space to measure the cross-modality similarity. However, previous methods often establish the shared embedding space based on linear mapping functions which might not be sophisticated enough to reveal more complicated inter-modal correspondences. Additionally, current studies assume that the rankings are of equal importance, and thus all rankings are used simultaneously, or a small number of rankings are selected randomly to train the embedding space at each iteration. Such strategies, however, always suffer from outliers as well as reduced generalization capability due to their lack of insightful understanding of procedure of human cognition. In this paper, we involve the self-paced learning theory with diversity into the cross-modal learning to rank and learn an optimal multi-modal embedding space based on non-linear mapping functions. This strategy enhances the model's robustness to outliers and achieves better generalization via training the model gradually from easy rankings by diverse queries to more complex ones. An efficient alternative algorithm is exploited to solve the proposed challenging problem with fast convergence in practice. Extensive experimental results on several benchmark datasets indicate that the proposed method achieves significant improvements over the state-of-the-arts in this literature.