mlsanityData to 7 Oct 2026

KnowShiftQA: How Robust are RAG Systems when Textbook Knowledge Shifts in K-12 Education?

arXiv 2412.08985 ↗First posted 12 Dec 2024cs.CLcited by 3

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems show remarkable potential as question answering tools in the K-12 Education domain, where knowledge is typically queried within the restricted scope of authoritative textbooks. However, discrepancies between these textbooks and the parametric knowledge inherent in Large Language Models (LLMs) can undermine the effectiveness of RAG systems. To systematically investigate RAG system robustness against such knowledge discrepancies, we introduce KnowShiftQA. This novel question answering dataset simulates these discrepancies by applying deliberate hypothetical knowledge updates to both answers and source documents, reflecting how textbook knowledge can shift. KnowShiftQA comprises 3,005 questions across five subjects, designed with a comprehensive question typology focusing on context utilization and knowledge integration. Our extensive experiments on retrieval and question answering performance reveal that most RAG systems suffer a substantial performance drop when faced with these knowledge discrepancies. Furthermore, questions requiring the integration of contextual (textbook) knowledge with parametric (LLM) knowledge pose a significant challenge to current LLMs.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 19 of its 53 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

19 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection17 Oct

Other references · 18

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  4. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  5. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  6. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  7. 2022Large Language Models with Controllable Working Memorycited by 169
  8. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  9. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  10. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  11. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  12. 2024Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extractioncited by 33
6 more
  1. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  2. 2024Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Informationcited by 20
  3. 2023Advancing Abductive Reasoning in Knowledge Graphs through Complex Logical Hypothesis Generationcited by 19
  4. 2023Sequential Query Encoding For Complex Query Answering on Knowledge Graphscited by 13
  5. 2024CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledgecited by 9
  6. 2019EDUQA: Educational Domain Question Answering System using Conceptual Network Mappingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025OnRL-RAG: Real-Time Personalized Mental Health Dialogue Systemcited by 1
  2. 2025Aligning LLMs for the Classroom with Knowledge-Based Retrieval -- A Comparative RAG Studycited by 1
  3. 2026OmniEdu: Open Foundation Models for Learning and Teachingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Machine Assistant with Reliable Knowledge: Enhancing Student Learning via RAG-based Retrieval
  2. 2025Answering Students' Questions on Course Forums Using Multiple Chain-of-Thought Reasoning and Finetuning RAG-Enabled LLM
  3. 2024Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generation
  4. 2025Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on a Math Textbook
  5. 2023Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference
  6. 2025Evaluating the Retrieval Robustness of Large Language Models
  7. 2024Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generation
  8. 2025Dynamic and Parametric Retrieval-Augmented Generation