# LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMs

arXiv: https://arxiv.org/abs/2309.00841 · HTML: https://mlsanity.com/paper/2309.00841/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 2 Sep 2023.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://mlsanity.com/paper/2302.08081/) (2023)
- [Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering](https://mlsanity.com/paper/2304.12102/) (2023)
- [Leveraging BERT for Extractive Text Summarization on Lectures](https://mlsanity.com/paper/1906.04165/) (2019)
- [Extractive Summarization via ChatGPT for Faithful Summary Generation](https://mlsanity.com/paper/2304.04193/) (2023)
- [An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT](https://mlsanity.com/paper/2304.08448/) (2023)
- [SummIt: Iterative Text Summarization via ChatGPT](https://mlsanity.com/paper/2305.14835/) (2023)
- [GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts](https://mlsanity.com/paper/2305.12477/) (2023)
- [Semantic Compression With Large Language Models](https://mlsanity.com/paper/2304.12512/) (2023)
- [Large Language Model Programs](https://mlsanity.com/paper/2305.05364/) (2023)
- [CQSumDP: A ChatGPT-Annotated Resource for Query-Focused Abstractive Summarization Based on Debatepedia](https://mlsanity.com/paper/2305.06147/) (2023)

## Built on by (most cited first, 30 of 40)

- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching](https://mlsanity.com/paper/2503.05179/) (2025)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [EcoAssistant: Using LLM Assistant More Affordably and Accurately](https://mlsanity.com/paper/2310.03046/) (2023)
- [When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.05690/) (2025)
- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidence](https://mlsanity.com/paper/2402.12174/) (2024)
- [RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects](https://mlsanity.com/paper/2501.18365/) (2025)
- [DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.05654/) (2024)
- [CigaR: Cost-efficient Program Repair with LLMs](https://mlsanity.com/paper/2402.06598/) (2024)
- [BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Models](https://mlsanity.com/paper/2403.18365/) (2024)
- [iRAG: Advancing RAG for Videos with an Incremental Approach](https://mlsanity.com/paper/2404.12309/) (2024)
- [Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges](https://mlsanity.com/paper/2401.08664/) (2023)
- [ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation](https://mlsanity.com/paper/2509.21730/) (2025)
- [Affordable Generative Agents](https://mlsanity.com/paper/2402.02053/) (2024)
- [VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning](https://mlsanity.com/paper/2406.14056/) (2024)
- [Rescuing the Unpoisoned: Efficient Defense against Knowledge Corruption Attacks on RAG Systems](https://mlsanity.com/paper/2511.01268/) (2025)
- [QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression](https://mlsanity.com/paper/2408.00274/) (2024)
- [XMainframe: A Large Language Model for Mainframe Modernization](https://mlsanity.com/paper/2408.04660/) (2024)
- [Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain](https://mlsanity.com/paper/2509.03787/) (2025)
- [Exploring Prompt Engineering: A Systematic Review with SWOT Analysis](https://mlsanity.com/paper/2410.12843/) (2024)
- [RedactOR: An LLM-Powered Framework for Automatic Clinical Data De-Identification](https://mlsanity.com/paper/2505.18380/) (2025)
- [Benchmarking Large Language Model Volatility](https://mlsanity.com/paper/2311.15180/) (2023)
- [Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression](https://mlsanity.com/paper/2408.15491/) (2024)
- [GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models](https://mlsanity.com/paper/2501.12956/) (2025)
- [Cost-efficient Knowledge-based Question Answering with Large Language Models](https://mlsanity.com/paper/2405.17337/) (2024)
- [A Convex-optimization-based Layer-wise Post-training Pruner for Large Language Models](https://mlsanity.com/paper/2408.03728/) (2024)
- [Evaluating ChatGPT on Nuclear Domain-Specific Data](https://mlsanity.com/paper/2409.00090/) (2024)

## Abstract (arXiv)

Question-answering (QA) is a significant application of Large Language Models (LLMs), shaping chatbot capabilities across healthcare, education, and customer service. However, widespread LLM integration presents a challenge for small businesses due to the high expenses of LLM API usage. Costs rise rapidly when domain-specific data (context) is used alongside queries for accurate domain-specific LLM responses. One option is to summarize the context by using LLMs and reduce the context. However, this can also filter out useful information that is necessary to answer some domain-specific queries. In this paper, we shift from human-oriented summarizers to AI model-friendly summaries. Our approach, LeanContext, efficiently extracts k key sentences from the context that are closely aligned with the query. The choice of k is neither static nor random; we introduce a reinforcement learning technique that dynamically determines k based on the query and context. The rest of the less important sentences are reduced using a free open source text reduction method. We evaluate LeanContext against several recent query-aware and query-unaware context reduction approaches on prominent datasets (arxiv papers and BBC news articles). Despite cost reductions of 37.29% to 67.81%, LeanContext's ROUGE-1 score decreases only by 1.41% to 2.65% compared to a baseline that retains the entire context (no summarization). Additionally, if free pretrained LLM-based summarizers are used to reduce context (into human consumable summaries), LeanContext can further modify the reduced context to enhance the accuracy (ROUGE-1 score) by 13.22% to 24.61%.