mlsanityData to 7 Oct 2026

ReToken: One Token to Improve Vision-Language Models for Visual Retrieval

arXiv 2607.28627 ↗First posted 30 Jul 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Long visual contexts challenge vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once can exceed GPU memory limits. We present RETOKEN, a single learnable embedding that extracts retrieval signals from the VLM's internal representations to select query-relevant visual tokens from the pre-filled KV cache. This enables retrieval within the answering VLM, without a separate retriever or re-encoding. Despite being trained on only a small image-QA dataset, RETOKEN generalizes across image and video benchmarks. On Visual Haystacks, it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative). On LVBench, it transfers zero-shot to long video and improves Qwen3VL-8B by 8.0 points. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 51 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

48 of 51 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2025Qwen3-VL Technical Reportcited by 2,534
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
36 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  4. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  5. 2025OpenAI GPT-5 System Cardcited by 910
  6. 2021Extract Free Dense Labels from CLIPcited by 580
  7. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  8. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  9. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  10. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  11. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  12. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  13. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  14. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  15. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  16. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  17. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  18. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  19. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  20. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  21. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  22. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  23. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  24. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  25. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  26. 2021WebQA: Multihop and Multimodal QAcited by 109
  27. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  28. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  29. 2023Grounded Question-Answering in Long Egocentric Videoscited by 62
  30. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  31. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  32. 2024Goldfish: Vision-Language Understanding of Arbitrarily Long Videoscited by 38
  33. 2023Answer Mining from a Pool of Images: Towards Retrieval-Based Visual Question Answeringcited by 10
  34. 2025TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Modelscited by 7
  35. 2025REN: Fast and Efficient Region Encodings from Patch-Based Image Encoderscited by 5
  36. 2024One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  2. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
  3. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
  4. 2026Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval
  5. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
  6. 2025TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inference
  7. 2026VETO: Video Efficient Token Optimization for Vision Language Models
  8. 2024ViCToR: Improving Visual Comprehension via Token Reconstruction for Pretraining LMMs