# Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs

arXiv: https://arxiv.org/abs/2402.12052 · HTML: https://mlsanity.com/paper/2402.12052/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 77 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 30 of 60)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)

## Built on by (most cited first, 30 of 34)

- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems](https://mlsanity.com/paper/2411.02959/) (2024)
- [Teola: Towards End-to-End Optimization of LLM-based Applications](https://mlsanity.com/paper/2407.00326/) (2024)
- [SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model](https://mlsanity.com/paper/2501.18636/) (2025)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation](https://mlsanity.com/paper/2412.11919/) (2024)
- [DualRAG: A Dual-Process Approach to Integrate Reasoning and Retrieval for Multi-Hop Question Answering](https://mlsanity.com/paper/2504.18243/) (2025)
- [SmartRAG: Jointly Learn RAG-Related Tasks From the Environment Feedback](https://mlsanity.com/paper/2410.18141/) (2024)
- [Hierarchical Document Refinement for Long-context Retrieval-augmented Generation](https://mlsanity.com/paper/2505.10413/) (2025)
- [Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization](https://mlsanity.com/paper/2410.02721/) (2024)
- [ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checking](https://mlsanity.com/paper/2510.13842/) (2025)
- [Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation](https://mlsanity.com/paper/2502.03233/) (2025)
- [C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.06205/) (2025)
- [CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation](https://mlsanity.com/paper/2410.23090/) (2024)
- [A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness](https://mlsanity.com/paper/2510.13890/) (2025)
- [One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.19670/) (2024)
- [Neuro-Symbolic Query Compiler](https://mlsanity.com/paper/2505.11932/) (2025)
- [GraphSearch: An Agentic Deep Searching Workflow for Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.22009/) (2025)
- [Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.02388/) (2025)
- [PaperAsk: A Benchmark for Reliability Evaluation of LLMs in Paper Search and Reading](https://mlsanity.com/paper/2510.22242/) (2025)
- [KAG-Thinker: Interactive Thinking and Deep Reasoning in LLMs via Knowledge-Augmented Generation](https://mlsanity.com/paper/2506.17728/) (2025)
- [MARS: Co-evolving Dual-System Deep Research via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.04935/) (2025)
- [Oreo: A Plug-in Context Reconstructor to Enhance Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.13019/) (2025)
- [Single LLM, Multiple Roles: A Unified Retrieval-Augmented Generation Framework Using Role-Specific Token Optimization](https://mlsanity.com/paper/2505.15444/) (2025)
- [Chain or tree? Re-evaluating complex reasoning from the perspective of a matrix of thought](https://mlsanity.com/paper/2509.03918/) (2025)
- [KCM: KAN-Based Collaboration Models Enhance Pretrained Large Models](https://mlsanity.com/paper/2510.20278/) (2025)

## Abstract (arXiv)

The integration of large language models (LLMs) and search engines represents a significant evolution in knowledge acquisition methodologies. However, determining the knowledge that an LLM already possesses and the knowledge that requires the help of a search engine remains an unresolved issue. Most existing methods solve this problem through the results of preliminary answers or reasoning done by the LLM itself, but this incurs excessively high computational costs. This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in LLMs with a slim proxy model, to enhance the LLM's knowledge acquisition process. We employ a proxy model which has far fewer parameters, and take its answers as heuristic answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM. We only conduct retrieval for the missing knowledge in questions that the LLM does not know. Extensive experimental results on five datasets with two LLMs demonstrate a notable improvement in the end-to-end performance of LLMs in question-answering tasks, achieving or surpassing current state-of-the-art models with lower LLM inference costs.