mlsanityData to 7 Oct 2026

Diffusion-Pretrained Dense and Contextual Embeddings

arXiv 2602.11151 ↗First posted 11 Feb 2026cs.LG cs.CL cs.IRcited by 9

Not on the map. Why?

Abstract (arXiv)

In this report, we introduce pplx-embed, a family of multilingual embedding models that employ multi-stage contrastive learning on a diffusion-pretrained language model backbone for web-scale retrieval. By leveraging bidirectional attention through diffusion-based pretraining, our models capture comprehensive bidirectional context within passages, enabling the use of mean pooling and a late chunking strategy to better preserve global context across long documents. We release two model types: pplx-embed-v1 for standard retrieval, and pplx-embed-context-v1 for contextualized embeddings that incorporate global document context into passage representations. pplx-embed-v1 achieves competitive performance on the MTEB(Multilingual, v2), MTEB(Code), MIRACL, BERGEN, and ToolRet retrieval benchmarks, while pplx-embed-context-v1 sets new records on the ConTEB benchmark. Beyond public benchmarks, pplx-embed-v1 demonstrates strong performance on our internal evaluation suite, focusing on real-world, large-scale search scenarios constructed from 1B production web pages. These results validate the models' effectiveness in production environments where retrieval quality and efficiency are critical at scale.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

42 of 54 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  8. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  10. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  11. 2025Large Language Diffusion Modelscited by 872
  12. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
30 more
  1. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  3. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  4. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  5. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  6. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  7. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  8. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  9. 2022Matryoshka Representation Learningcited by 268
  10. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  11. 2024Generative Representational Instruction Tuningcited by 227
  12. 2024Scaling up Masked Diffusion Models on Textcited by 195
  13. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
  14. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
  15. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 120
  16. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  17. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  18. 2024Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Modelscited by 70
  19. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  20. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
  21. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  22. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24
  23. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspectivecited by 23
  24. 2024Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?cited by 19
  25. 2024Contextual Document Embeddingscited by 19
  26. 2025Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMscited by 17
  27. 2025Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddingscited by 12
  28. 2024Little Giants: Synthesizing High-Quality Embedding Data at Scalecited by 11
  29. 2024Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipecited by 4
  30. 2025Optimization of embeddings storage for RAG systems using quantization and dimensionality reduction techniquescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agentscited by 37
  2. 2026DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Searchcited by 3
  3. 2026Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agentscited by 2
  4. 2026On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stabilitycited by 0
  5. 2026DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Modelscited by 0
  6. 2026Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performancecited by 0
  7. 2026Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systemscited by 0
  8. 2026Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systemscited by 0
  9. 2026QureRadEmbed: Structuring Radiological Similarity through Attribute and Reasoning Supervisioncited by 0