mlsanityData to 14 Oct 2026

CUB: Benchmarking Context Utilisation Techniques for Language Models

arXiv 2505.16518 ↗First posted 22 May 2025cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Incorporating external knowledge is crucial for knowledge-intensive tasks, such as question answering and fact checking. However, language models (LMs) may ignore relevant information that contradicts outdated parametric memory or be distracted by irrelevant contexts. While many context utilisation manipulation techniques (CMTs) have recently been proposed to alleviate these issues, few have seen systematic comparison. In this paper, we develop CUB (Context Utilisation Benchmark) - the first comprehensive benchmark designed to help diagnose CMTs under diverse noisy context conditions within retrieval-augmented generation (RAG). With this benchmark, we conduct the most extensive evaluation to date of seven state-of-the-art methods, representative of the main categories of CMTs, across three diverse datasets and tasks, applied to 11 LMs. Our findings expose critical gaps in current CMT evaluation practices, demonstrating the need for holistic testing. We reveal that most existing CMTs struggle to handle the full spectrum of context types encountered in real-world RAG scenarios. We also find that many CMTs display inflated performance on simple synthesised datasets, compared to more realistic datasets with naturally occurring samples.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 45 of its 68 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

45 of 68 references.

References · 45

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2019Language Models as Knowledge Bases?cited by 1,936
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  11. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
33 more
  1. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  2. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  3. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  4. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  5. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  6. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  7. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  8. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  9. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  10. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  11. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  12. 2022Large Language Models with Controllable Working Memorycited by 169
  13. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  14. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 142
  15. 2024Large Language Models Can Learn Temporal Reasoningcited by 111
  16. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 91
  17. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 89
  18. 2024What Evidence Do Language Models Find Convincing?cited by 75
  19. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  20. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 63
  21. 2023Scientific Fact-Checking: A Survey of Resources and Approachescited by 54
  22. 2024Enhancing Contextual Understanding in Large Language Models through Contrastive Decodingcited by 49
  23. 2024Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactualscited by 40
  24. 2024AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledgecited by 37
  25. 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
  26. 2024Discerning and Resolving Knowledge Conflicts through Adaptive Decoding with Contextual Information-Entropy Constraintcited by 27
  27. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  28. 2022TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Mediacited by 19
  29. 2024Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contextscited by 19
  30. 2023The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Modelscited by 11
  31. 2024Desiderata for the Context Use of Question Answering Systemscited by 7
  32. 2024Fact Recall, Heuristics or Pure Guesswork? Precise Interpretations of Language Models for Fact Completioncited by 4
  33. 2025Language Model Re-rankers are Fooled by Lexical Similaritiescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Understanding the Interplay between LLMs' Utilisation of Parametric and Contextual Knowledge: A keynote at ECIR 2025cited by 1
  2. 2026Response-Aware User Memory Selection for LLM Personalizationcited by 1
  3. 2026The Compliance Trap: Diagnosing How AI Agents Consume Conflicting Memorycited by 1
  4. 2025Evaluation Framework for Highlight Explanations of Context Utilisation in Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing
  2. 2026Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
  3. 2026CL-bench: A Benchmark for Context Learning
  4. 2026Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
  5. 2024From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries
  6. 2025On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generation
  7. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
  8. 2025Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation