mlsanityData to 14 Oct 2026

Very Efficient Listwise Multimodal Reranking for Long Documents

arXiv 2605.11864 ↗First posted 12 May 2026cs.IR cs.AI cs.CVcited by 2

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Listwise reranking is a key yet computationally expensive component in vision-centric retrieval and multimodal retrieval-augmented generation (M-RAG) over long documents. While recent VLM-based rerankers achieve strong accuracy, their practicality is often limited by long visual-token sequences and multi-step autoregressive decoding. We propose ZipRerank, a highly efficient listwise multimodal reranker that directly addresses both bottlenecks. It reduces input length via a lightweight query-image early interaction mechanism and eliminates autoregressive decoding by scoring all candidates in a single forward pass. To enable effective learning, ZipRerank adopts a two-stage training strategy: (i) listwise pretraining on large-scale text data rendered as images, and (ii) multimodal finetuning with VLM-teacher-distilled soft-ranking supervision. Extensive experiments on the MMDocIR benchmark show that ZipRerank matches or surpasses state-of-the-art multimodal rerankers while reducing LLM inference latency by up to an order of magnitude, making it well-suited for latency-sensitive real-world systems. The code is available at https://github.com/dukesun99/ZipRerank.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 62 of its 83 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

62 of 83 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers28 May

Other references · 61

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2025Qwen3-VL Technical Reportcited by 2,793
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
38 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2019Passage Re-ranking with BERTcited by 645
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  4. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  6. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  7. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  8. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  9. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  10. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  11. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  12. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 155
  13. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 135
  14. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  15. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 110
  16. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  17. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
  18. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  19. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 85
  20. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  21. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 79
  22. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  23. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
  24. 2025ReasonRank: Empowering Passage Ranking with Strong Reasoning Abilitycited by 42
  25. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  26. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  27. 2025DIVER: A Multi-Stage Approach for Reasoning-intensive Information Retrievalcited by 37
  28. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  29. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
  30. 2025Guiding Retrieval using LLM-based Listwise Rankerscited by 16
  31. 2025RankLLM: A Python Package for Reranking with LLMscited by 16
  32. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13
  33. 2020Locality-Sensitive Hashing Scheme based on Longest Circular Co-Substringcited by 11
  34. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generationcited by 11
  35. 2024A General Framework for Producing Interpretable Semantic Text Embeddingscited by 10
  36. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  37. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  38. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 9

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankerscited by 0
  2. 2026AMBER: Multi-View Adaptive Budget Allocation for Listwise Vision-Language Rerankingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Vision Meets Texts in Listwise Reranking
  2. 2026ResRank: Unifying Retrieval and Listwise Reranking via End-to-End Joint Training with Residual Passage Compression
  3. 2026miniReranker: Efficient Multimodal Reranking through Visual Cache Reuse and Interaction Sparsity
  4. 2026Efficient Listwise Reranking with Compressed Document Representations
  5. 2026UniRank: End-to-End Domain-Specific Reranking of Hybrid Text-Image Candidates
  6. 2026PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization
  7. 2026RidgeRank: Efficient Visual Document Reranking via Score Fusion and a Shallow Linear Readout
  8. 2026Indexing Multimodal Language Models for Large-scale Image Retrieval