# Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering

arXiv: https://arxiv.org/abs/2304.12102 · HTML: https://mlsanity.com/paper/2304.12102/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)

## Built on by (most cited first, 30 of 65)

- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [Chain of Agents: Large Language Models Collaborating on Long-Context Tasks](https://mlsanity.com/paper/2406.02818/) (2024)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research](https://mlsanity.com/paper/2405.13576/) (2024)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Parrot: Efficient Serving of LLM-based Applications with Semantic Variable](https://mlsanity.com/paper/2405.19888/) (2024)
- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction](https://mlsanity.com/paper/2312.12343/) (2023)
- [Extending Context Window of Large Language Models via Semantic Compression](https://mlsanity.com/paper/2312.09571/) (2023)
- [Context as a Tool: Context Management for Long-Horizon SWE-Agents](https://mlsanity.com/paper/2512.22087/) (2025)
- [Discrete Prompt Compression with Reinforcement Learning](https://mlsanity.com/paper/2308.08758/) (2023)
- [LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMs](https://mlsanity.com/paper/2309.00841/) (2023)
- [HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems](https://mlsanity.com/paper/2411.02959/) (2024)
- [The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey](https://mlsanity.com/paper/2401.07872/) (2024)
- [BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidence](https://mlsanity.com/paper/2402.12174/) (2024)
- [Context Embeddings for Efficient Answer Generation in RAG](https://mlsanity.com/paper/2407.09252/) (2024)
- [DualRAG: A Dual-Process Approach to Integrate Reasoning and Retrieval for Multi-Hop Question Answering](https://mlsanity.com/paper/2504.18243/) (2025)
- [Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression](https://mlsanity.com/paper/2404.00489/) (2024)
- [SpreadsheetLLM: Encoding Spreadsheets for Large Language Models](https://mlsanity.com/paper/2407.09025/) (2024)
- [Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Bias](https://mlsanity.com/paper/2407.15366/) (2024)
- [An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.01549/) (2024)
- [Understanding and Improving Information Preservation in Prompt Compression for LLMs](https://mlsanity.com/paper/2503.19114/) (2025)
- [E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning](https://mlsanity.com/paper/2409.06679/) (2024)
- [Keyword Augmented Retrieval: Novel framework for Information Retrieval integrated with speech interface](https://mlsanity.com/paper/2310.04205/) (2023)
- [Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation](https://mlsanity.com/paper/2409.15895/) (2024)
- [Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression](https://mlsanity.com/paper/2402.16058/) (2024)
- [SelfCP: Compressing Over-Limit Prompt via the Frozen Large Language Model Itself](https://mlsanity.com/paper/2405.17052/) (2024)
- [AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees](https://mlsanity.com/paper/2512.04550/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have received significant attention by achieving remarkable performance across various tasks. However, their fixed context length poses challenges when processing long documents or maintaining extended conversations. This paper proposes a method called textitSelective Context that employs self-information to filter out less informative content, thereby enhancing the efficiency of the fixed context length. We demonstrate the effectiveness of our approach on tasks of summarisation and question answering across different data sources, including academic papers, news articles, and conversation transcripts.