# MINER: Mining Multimodal Internal Representation for Efficient Retrieval

arXiv: https://arxiv.org/abs/2605.06460 · HTML: https://mlsanity.com/paper/2605.06460/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 36 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 30 of 31)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking](https://mlsanity.com/paper/2601.04720/) (2026)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [ColPali: Efficient Document Retrieval with Vision Language Models](https://mlsanity.com/paper/2407.01449/) (2024)
- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [Unifying Multimodal Retrieval via Document Screenshot Embedding](https://mlsanity.com/paper/2406.11251/) (2024)
- [Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning](https://mlsanity.com/paper/2201.03529/) (2022)
- [jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval](https://mlsanity.com/paper/2506.18902/) (2025)
- [ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval](https://mlsanity.com/paper/2505.17166/) (2025)
- [Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2212.03220/) (2022)
- [Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering](https://mlsanity.com/paper/2309.17133/) (2023)
- [MMDocIR: Benchmarking Multimodal Retrieval for Long Documents](https://mlsanity.com/paper/2501.08828/) (2025)
- [Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings](https://mlsanity.com/paper/2506.04997/) (2025)
- [ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios](https://mlsanity.com/paper/2601.08620/) (2026)
- [Deep Learning based Visually Rich Document Content Understanding: A Survey](https://mlsanity.com/paper/2408.01287/) (2024)
- [MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings](https://mlsanity.com/paper/2506.23115/) (2025)
- [VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding](https://mlsanity.com/paper/2508.07493/) (2025)
- [Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization](https://mlsanity.com/paper/2510.05038/) (2025)
- [LLM Safety From Within: Detecting Harmful Content with Internal Representations](https://mlsanity.com/paper/2604.18519/) (2026)
- [SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classification](https://mlsanity.com/paper/2311.15983/) (2023)
- [Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding](https://mlsanity.com/paper/2510.15253/) (2025)
- [Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach](https://mlsanity.com/paper/2407.06964/) (2024)
- [Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction](https://mlsanity.com/paper/2505.07730/) (2025)

## Built on by (most cited first, 2 of 2)

- [DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation](https://mlsanity.com/paper/2608.10636/) (2026)
- [Agent Safety From Within: Detecting Harmful Trajectories from LLM Internal States](https://mlsanity.com/paper/2609.33039/) (2026)

## Abstract (arXiv)

Visual document retrieval has become essential for accessing information in visually rich documents. Existing approaches fall into two camps. Late-interaction retrievers achieve strong quality through fine-grained token-level matching but store hundreds of vectors per page, incurring large index footprints and high serving costs. By contrast, dense single-vector retrievers retain storage and latency advantages but consistently lag in quality because they compress all information into a single final-layer embedding. In this work, we first conduct a layerwise diagnostic on single-vector retrievers, revealing that retrieval-relevant signal resides in internal representations. Motivated by these findings, we propose MINER (Mining Multimodal Internal RepreseNtation for Efficient Retrieval), a lightweight plug-in module that probes and fuses internal signals across transformer layers into a single compact embedding without modifying the backbone or sacrificing single-vector efficiency. The first Retrieval-Aligned Layer Probing stage attaches a lightweight probe at each layer, surfacing which dimensions carry retrieval-relevant information. The subsequent Adaptive Sparse Multi-Layer Fusion stage applies performance-adaptive neuron-level masking to the selected layers and fuses the surviving signals into the final dense vector. Across ViDoRe V1/V2/V3, MINER outperforms existing dense single-vector retrievers on the majority of benchmarks, with up to 4.5% nDCG@5 improvement over its corresponding backbone. Compared to strong late-interaction baselines, in some settings MINER substantially narrows the nDCG@5 gap to 0.2 while preserving the storage and serving advantages of dense retrieval.