mlsanityData to 7 Oct 2026

DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models

arXiv 2605.07210 ↗First posted 8 May 2026cs.IR cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

This paper shows how diffusion language models (DLMs) can be used as effective and efficient retrievers. Existing DLM-based retrievers (e.g., DiffEmbed) follow BERT-style encoding, representing each query or passage as a single mean-pooled vector. This ignores how DLMs are trained to generate responses through masked-position prediction under bidirectional attention, a capability that can provide stronger retrieval signals. We propose DiffRetriever, which uses the DLM's native masked-position prediction directly for retrieval. For each query or passage, DiffRetriever appends one or more masked positions, using the outputs as retrieval representations in a single forward pass. With one masked position, single-representation DiffRetriever already improves over DiffEmbed on the same backbones. DiffRetriever also naturally extends to multi-representation retrieval: DLMs process multiple masked positions jointly, enabling ColBERT-style fine-grained matching with little additional encoding latency. In autoregressive LLM retrievers, the same multi-representation strategy requires sequential decoding and therefore incurs much higher latency. DiffRetriever obtains the strongest aggregate effectiveness within our matched comparison, outperforming DiffEmbed, PromptReps, and RepLLaMA. Masked-position counts selected on training data transfer well across datasets, while per-query variation suggests headroom for adaptive allocation. Code is available at https://github.com/ielab/diffretriever.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 22 of its 27 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

22 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026DiffuRank: Effective Document Reranking with Diffusion Language Models13 Feb
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025Large Language Diffusion Models14 Feb

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  4. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  7. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  9. 2023Improving Text Embeddings with Large Language Modelscited by 295
  10. 2020Overview of the TREC 2019 deep learning trackcited by 282
  11. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  12. 2021Overview of the TREC 2020 deep learning trackcited by 193
7 more
  1. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  2. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  3. 2024PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrievalcited by 30
  4. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspectivecited by 23
  5. 2022To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrieverscited by 12
  6. 2026Diffusion-Pretrained Dense and Contextual Embeddingscited by 9
  7. 2024An Investigation of Prompt Variations for Zero-shot LLM-based Rankerscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Models
  2. 2025DiffuGR: Generative Document Retrieval with Diffusion Language Models
  3. 2025A Survey on Diffusion Language Models
  4. 2026Self-Augmenting Retrieval for Diffusion Language Models
  5. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  6. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
  7. 2026Improving Diffusion Language Model Decoding through Joint Search in Generation Order and Token Space
  8. 2026Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching