# Bottleneck Tokens for Unified Multimodal Retrieval

arXiv: https://arxiv.org/abs/2604.11095 · HTML: https://mlsanity.com/paper/2604.11095/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 27 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [ColPali: Efficient Document Retrieval with Vision Language Models](https://mlsanity.com/paper/2407.01449/) (2024)
- [GME: Improving Universal Multimodal Retrieval by Multimodal LLMs](https://mlsanity.com/paper/2412.16855/) (2024)
- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [E5-V: Universal Embeddings with Multimodal Large Language Models](https://mlsanity.com/paper/2407.12580/) (2024)
- [VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents](https://mlsanity.com/paper/2507.04590/) (2025)
- [LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant](https://mlsanity.com/paper/2412.01720/) (2024)
- [UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning](https://mlsanity.com/paper/2510.13515/) (2025)
- [Think Then Embed: Generative Context Improves Multimodal Embedding](https://mlsanity.com/paper/2510.05014/) (2025)
- [UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings](https://mlsanity.com/paper/2511.00405/) (2025)
- [CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning](https://mlsanity.com/paper/2503.19900/) (2025)
- [RzenEmbed: Towards Comprehensive Multimodal Retrieval](https://mlsanity.com/paper/2510.27350/) (2025)
- [Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings](https://mlsanity.com/paper/2602.13823/) (2026)

## Built on by (most cited first, 2 of 2)

- [UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval](https://mlsanity.com/paper/2608.18504/) (2026)
- [Autoregressive Retriever: Improving Query Understanding from Item Feedback for Universal Multimodal Retrieval](https://mlsanity.com/paper/2610.11666/) (2026)

## Abstract (arXiv)

Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA).