# Directed Information γ-covering: An Information-Theoretic Framework for Context Engineering

arXiv: https://arxiv.org/abs/2510.00079 · HTML: https://mlsanity.com/paper/2510.00079/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions](https://mlsanity.com/paper/2503.23278/) (2025)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [SirLLM: Streaming Infinite Retentive LLM](https://mlsanity.com/paper/2405.12528/) (2024)
- [The information bottleneck and geometric clustering](https://mlsanity.com/paper/1712.09657/) (2017)
- [Information-Theoretic Distillation for Reference-less Summarization](https://mlsanity.com/paper/2403.13780/) (2024)
- [Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation](https://mlsanity.com/paper/2411.07773/) (2024)
- [Investigating Entropy for Extractive Document Summarization](https://mlsanity.com/paper/2109.10886/) (2021)

## Built on by (most cited first, 1 of 1)

- [Gated Differentiable Working Memory for Long-Context Language Modeling](https://mlsanity.com/paper/2601.12906/) (2026)

## Abstract (arXiv)

We introduce Directed Information γ-covering, a simple but general framework for redundancy-aware context engineering. Directed information (DI), a causal analogue of mutual information, measures asymmetric predictiveness between chunks. If operatornameDIi → j ≥ H(Cj) - γ, then Ci suffices to represent Cj up to γ bits. Building on this criterion, we formulate context selection as a γ-cover problem and propose a greedy algorithm with provable guarantees: it preserves query information within bounded slack, inherits (1+ln n) and (1-1/e) approximations from submodular set cover, and enforces a diversity margin. Importantly, building the γ-cover is query-agnostic: it incurs no online cost and can be computed once offline and amortized across all queries. Experiments on HotpotQA show that γ-covering consistently improves over BM25, a competitive baseline, and provides clear advantages in hard-decision regimes such as context compression and single-slot prompt selection. These results establish DI γ-covering as a principled, self-organizing backbone for modern LLM pipelines.