mlsanityData to 7 Oct 2026

UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation

arXiv 2410.02719 ↗First posted 3 Oct 2024cs.CLcited by 14

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

We present UncertaintyRAG, a novel approach for long-context Retrieval-Augmented Generation (RAG) that utilizes Signal-to-Noise Ratio (SNR)-based span uncertainty to estimate similarity between text chunks. This span uncertainty enhances model calibration, improving robustness and mitigating semantic inconsistencies introduced by random chunking. Leveraging this insight, we propose an efficient unsupervised learning technique to train the retrieval model, alongside an effective data sampling and scaling strategy. UncertaintyRAG outperforms baselines by 2.03% on LLaMA-2-7B, achieving state-of-the-art results while using only 4% of the training data compared to other advanced open-source retrieval models under distribution shift settings. Our method demonstrates strong calibration through span uncertainty, leading to improved generalization and robustness in long-context RAG tasks. Additionally, UncertaintyRAG provides a lightweight retrieval model that can be integrated into any large language model with varying context window lengths, without the need for fine-tuning, showcasing the flexibility of our approach.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 76 of its 85 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

76 of 85 references.

References · 76

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
38 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  13. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  14. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  15. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  16. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  17. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  18. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  19. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  20. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  21. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  22. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  23. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  24. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  25. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  26. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  27. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  28. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  29. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  30. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  31. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  32. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  33. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  34. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  35. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  36. 2023Augmenting Language Models with Long-Term Memorycited by 186
  37. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  38. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
  3. 2025FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectorscited by 19
  4. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  5. 2025SearchRAG: Can Search Engines Be Helpful for LLM-based Medical Question Answering?cited by 9
  6. 2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 4
  7. 2025LLM-Centric RAG with Multi-Granular Indexing and Confidence Constraintscited by 3
  8. 2026Predict the Retrieval! Test time adaptation for Retrieval Augmented Generationcited by 2
  9. 2026IV-CoT: Implicit Visual Chain-of-Thought for Structure-Aware Text-to-Image Generationcited by 2
  10. 2025Memory as Resonance: A Biomimetic Architecture for Infinite Context Memory on Ergodic Phonetic Manifoldscited by 1
  11. 2026Towards Dependable Retrieval-Augmented Generation Using Factual Confidence Predictioncited by 1
  12. 2026Adaptive Data Augmentation with Multi-armed Bandit: Sample-Efficient Embedding Calibration for Implicit Pattern Recognitioncited by 0
2 more
  1. 2026Did You Check the Right Pocket? Cost-Sensitive Store Routing for Memory-Augmented Agentscited by 0
  2. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Models
  2. 2026Uncertainty-Aware Hybrid Retrieval for Long-Document RAG
  3. 2025Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation
  4. 2026Uncertainty Quantification for Multimodal Retrieval Augmented Generation
  5. 2025SUGAR: Leveraging Contextual Confidence for Smarter Retrieval
  6. 2025MacRAG: Compress, Slice, and Scale-up for Multi-Scale Adaptive Context RAG
  7. 2025Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
  8. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey