# Think Then Embed: Generative Context Improves Multimodal Embedding

arXiv: https://arxiv.org/abs/2510.05014 · HTML: https://mlsanity.com/paper/2510.05014/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 36 of its 40 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [Query Rewriting for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2305.14283/) (2023)
- [UniIR: Training and Benchmarking Universal Multimodal Information Retrievers](https://mlsanity.com/paper/2311.17136/) (2023)
- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [jina-embeddings-v3: Multilingual Embeddings With Task LoRA](https://mlsanity.com/paper/2409.10173/) (2024)
- [Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup](https://mlsanity.com/paper/2101.06983/) (2021)
- [VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents](https://mlsanity.com/paper/2507.04590/) (2025)
- [MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval](https://mlsanity.com/paper/2412.14475/) (2024)
- [Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs](https://mlsanity.com/paper/2504.17432/) (2025)
- [Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting](https://mlsanity.com/paper/2310.09716/) (2023)
- [LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning](https://mlsanity.com/paper/2503.04812/) (2025)
- [mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data](https://mlsanity.com/paper/2502.08468/) (2025)
- [CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning](https://mlsanity.com/paper/2503.19900/) (2025)
- [Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining](https://mlsanity.com/paper/2505.11293/) (2025)
- [Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying](https://mlsanity.com/paper/2506.02020/) (2025)
- [MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings](https://mlsanity.com/paper/2506.23115/) (2025)
- [A Surprisingly Simple yet Effective Multi-Query Rewriting Method for Conversational Passage Retrieval](https://mlsanity.com/paper/2406.18960/) (2024)
- [ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2505.15776/) (2025)

## Built on by (most cited first, 30 of 41)

- [Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings](https://mlsanity.com/paper/2602.13823/) (2026)
- [PLUME: Latent Reasoning Based Universal Multimodal Embedding](https://mlsanity.com/paper/2604.02073/) (2026)
- [LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval](https://mlsanity.com/paper/2603.01425/) (2026)
- [ReMatch: Boosting Representation through Matching for Multimodal Retrieval](https://mlsanity.com/paper/2511.19278/) (2025)
- [WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report](https://mlsanity.com/paper/2608.24053/) (2026)
- [CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding](https://mlsanity.com/paper/2601.21262/) (2026)
- [DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories](https://mlsanity.com/paper/2602.10809/) (2026)
- [TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings](https://mlsanity.com/paper/2603.04772/) (2026)
- [MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control](https://mlsanity.com/paper/2604.06156/) (2026)
- [Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings](https://mlsanity.com/paper/2604.22280/) (2026)
- [LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck](https://mlsanity.com/paper/2606.13061/) (2026)
- [MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understanding](https://mlsanity.com/paper/2604.00513/) (2026)
- [Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs](https://mlsanity.com/paper/2602.05275/) (2026)
- [ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge](https://mlsanity.com/paper/2602.09839/) (2026)
- [MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning](https://mlsanity.com/paper/2603.03379/) (2026)
- [Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture](https://mlsanity.com/paper/2605.14448/) (2026)
- [TTE-Flash: Accelerating Reasoning-based Multimodal Representations via Think-Then-Embed Tokens](https://mlsanity.com/paper/2605.16638/) (2026)
- [PhotoCraft: Agentic Reasoning with Hierarchical Self-Evolving Memory for Deep Image Search](https://mlsanity.com/paper/2606.03099/) (2026)
- [Pailitao-VL: Unified Embedding and Reranker for Real-Time Multi-Modal Industrial Search](https://mlsanity.com/paper/2602.13704/) (2026)
- [Bottleneck Tokens for Unified Multimodal Retrieval](https://mlsanity.com/paper/2604.11095/) (2026)
- [Douyin Multimodal Embedding Model Technical Report](https://mlsanity.com/paper/2608.02148/) (2026)
- [UEmbed: Unified Sparse and Dense Multimodal Embeddings](https://mlsanity.com/paper/2608.02583/) (2026)
- [Illuminating Visual Identity in Universal Multimodal Embeddings](https://mlsanity.com/paper/2608.01794/) (2026)
- [UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval](https://mlsanity.com/paper/2608.18504/) (2026)
- [Learning Multimodal Embeddings with Evidence-Aligned Readout](https://mlsanity.com/paper/2609.33659/) (2026)
- [Adapting MLLMs for Nuanced Video Retrieval](https://mlsanity.com/paper/2512.13511/) (2025)
- [SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2601.04824/) (2026)
- [CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval](https://mlsanity.com/paper/2604.15663/) (2026)
- [Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding](https://mlsanity.com/paper/2606.09331/) (2026)
- [SHIFT: Self-reconstruction Harnesses Implicit Fine-grained Thinking for Retrieval](https://mlsanity.com/paper/2607.21333/) (2026)

## Abstract (arXiv)

There is a growing interest in Universal Multimodal Embeddings (UME), where models are required to generate task-specific representations. While recent studies show that Multimodal Large Language Models (MLLMs) perform well on such tasks, they treat MLLMs solely as encoders, overlooking their generative capacity. However, such an encoding paradigm becomes less effective as instructions become more complex and require compositional reasoning. Inspired by the proven effectiveness of chain-of-thought reasoning, we propose a general Think-Then-Embed (TTE) framework for UME, composed of a reasoner and an embedder. The reasoner MLLM first generates reasoning traces that explain complex queries, followed by an embedder that produces representations conditioned on both the original query and the intermediate reasoning. This explicit reasoning step enables more nuanced understanding of complex multimodal instructions. Our contributions are threefold. First, by leveraging a powerful MLLM reasoner, we achieve state-of-the-art performance on the MMEB-V2 benchmark, surpassing proprietary models trained on massive in-house datasets. Second, to reduce the dependency on large MLLM reasoners, we finetune a smaller MLLM reasoner using high-quality embedding-centric reasoning traces, achieving the best performance among open-source models with a 7% absolute gain over recently proposed models. Third, we investigate strategies for integrating the reasoner and embedder into a unified model for improved efficiency without sacrificing performance.