Directed Information γ-covering: An Information-Theoretic Framework for Context Engineering

We introduce Directed Information γ-covering, a simple but general framework for redundancy-aware context engineering. Directed information (DI), a causal analogue of mutual information, measures asymmetric predictiveness between chunks. If operatornameDIi → j ≥ H(Cj) - γ, then Ci suffices to represent Cj up to γ bits. Building on this criterion, we formulate context selection as a γ-cover problem and propose a greedy algorithm with provable guarantees: it preserves query information within bounded slack, inherits (1+ln n) and (1-1/e) approximations from submodular set cover, and enforces a diversity margin. Importantly, building the γ-cover is query-agnostic: it incurs no online cost and can be computed once offline and amortized across all queries. Experiments on HotpotQA show that γ-covering consistently improves over BM25, a competitive baseline, and provides clear advantages in hard-decision regimes such as context compression and single-slot prompt selection. These results establish DI γ-covering as a principled, self-organizing backbone for modern LLM pipelines.