# CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling

arXiv: https://arxiv.org/abs/2406.17507 · HTML: https://mlsanity.com/paper/2406.17507/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2024.

## Built on by (most cited first, 5 of 5)

- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval](https://mlsanity.com/paper/2603.26683/) (2026)
- [VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents](https://mlsanity.com/paper/2607.24748/) (2026)
- [Generative Universal Multimodal Retrieval with Dual-role Identifiers](https://mlsanity.com/paper/2608.12987/) (2026)
- [WIDE: Wildcard Inference with Dynamic Expansion for Cross-Modal Generative Retrieval](https://mlsanity.com/paper/2609.03554/) (2026)

## Abstract (arXiv)

Cross-modal retrieval aims to search for instances, which are semantically related to the query through the interaction of different modal data. Traditional solutions utilize a single-tower or dual-tower framework to explicitly compute the score between queries and candidates, which is challenged by training cost and inference latency with large-scale data. Inspired by the remarkable performance and efficiency of generative models, we propose a generative cross-modal retrieval framework (CART) based on coarse-to-fine semantic modeling, which assigns identifiers to each candidate and treats the generating identifier as the retrieval target. Specifically, we explore an effective coarse-to-fine scheme, combining K-Means and RQ-VAE to discretize multimodal data into token sequences that support autoregressive generation. Further, considering the lack of explicit interaction between queries and candidates, we propose a feature fusion strategy to align their semantics. Extensive experiments demonstrate the effectiveness of the strategies in the CART, achieving excellent results in both retrieval performance and efficiency.