mlsanityData to 14 Oct 2026

Let your LLM generate a few tokens and you will reduce the need for retrieval

arXiv 2412.11536 ↗First posted 16 Dec 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate how efficiently large language models (LLM) can be trained to check whether an answer is already stored in their parametric memory. We distill an LLM-as-a-judge to compute the IK (I Know) score. We found that this method is particularly beneficial in the context of retrieval-assisted augmented generation (RAG), with a respectable accuracy of 80%. It enables a significant reduction (more than 50%) in the number of search and reranking steps required for certain data sets. We have also introduced the IK score, which serves as a useful tool for characterising datasets by facilitating the classification task. Interestingly, through the inclusion of response tokens as input, our results suggest that only about 20,000 training samples are required to achieve good performance. The central element of this work is the use of a teacher model - the LLM as a judge - to generate training data. We also assess the robustness of the IK classifier by evaluating it with various types of teachers, including both string-based methods and LLMs, with the latter providing better results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

15 of 21 references.

References · 15

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  6. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  7. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  8. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  9. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  10. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  11. 2019Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Indexcited by 64
  12. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
3 more
  1. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  2. 2024Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMscited by 34
  3. 2024When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectivelycited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Tuning LLMs by RAG Principles: Towards LLM-native Memory
  2. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  3. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs
  4. 2025Post-training an LLM for RAG? Train on Self-Generated Demonstrations
  5. 2024One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models
  6. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models
  7. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely
  8. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems