# An Information Theoretic Perspective on Agentic System Design arXiv: https://arxiv.org/abs/2512.21720 · HTML: https://mlsanity.com/paper/2512.21720/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. - Builds on 52 of its 68 references that are in the map. - First posted on arXiv 25 Dec 2025. ## Placed in - LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/) ## Builds on (most cited first, 30 of 52) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021) - [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025) - [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021) - [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018) - [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024) - [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023) - [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016) - [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015) - [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024) - [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023) - [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019) - [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023) - [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017) - [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021) - [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024) - [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024) - [Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://mlsanity.com/paper/2510.04618/) (2025) - [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023) - [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023) - [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025) - [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025) - [MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents](https://mlsanity.com/paper/2506.15841/) (2025) - [Chain of Agents: Large Language Models Collaborating on Long-Context Tasks](https://mlsanity.com/paper/2406.02818/) (2024) - [Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2503.09572/) (2025) - [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018) ## Built on by (most cited first, 8 of 8) - [Phase Transition for Budgeted Multi-Agent Synergy](https://mlsanity.com/paper/2601.17311/) (2026) - [When Does Multi-Agent Collaboration Help? An Entropy Perspective](https://mlsanity.com/paper/2602.04234/) (2026) - [Cognitive Models and AI Algorithms Provide Templates for Designing Language Agents](https://mlsanity.com/paper/2602.22523/) (2026) - [QUIVER: A Formal Framework for Quantifying Perturbation Propagation and Bifurcation in Compound AI Systems](https://mlsanity.com/paper/2605.23956/) (2026) - [Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Model](https://mlsanity.com/paper/2602.01778/) (2026) - [From Unstructured Recall to Schema-Grounded Memory: Reliable AI Memory via Iterative, Schema-Aware Extraction](https://mlsanity.com/paper/2604.27906/) (2026) - [The Compaction Cliff in Long-Running AI Agent Memory](https://mlsanity.com/paper/2608.22752/) (2026) - [The Communication Bottleneck: A Round-Trip Study of Tree-Structured Expression Serialization in Language Models](https://mlsanity.com/paper/2609.21509/) (2026) ## Abstract (arXiv) Agentic language model (LM) systems power modern applications like "Deep Research" and "Claude Code," and leverage multi-LM architectures to overcome context limitations. Beneath their apparent diversity lies a recurring pattern: smaller "compressor" LMs (that can even run locally) distill raw context into compact text that is then consumed by larger "predictor" LMs. Despite their popularity, the design of compressor-predictor systems remains largely ad hoc, with little guidance on how compressor and predictor choices shape downstream performance. In practice, attributing gains to compression versus prediction requires costly, task-specific pairwise sweeps. We argue that these agentic system design questions are, at root, information-theoretic. Viewing the compressor LM as a noisy channel, we introduce a simple estimator of mutual information between the context and its compression to quantify compression quality in a task-independent way. We show that mutual information strongly predicts downstream performance, independent of any specific task. Through an information-theoretic framework, we perform a comprehensive empirical analysis across five datasets and three model families. Results reveal that larger compressors not only are more accurate, but also more token-efficient, conveying more bits of information per token. A 7B Qwen-2.5 compressor, for instance, is 1.6× more accurate, 4.6× more concise, and conveys 5.5× more bits of mutual information per token than its 1.5B sibling. Across datasets, scaling compressors is substantially more effective than scaling predictors, enabling larger on-device compressors to pair with smaller cloud predictors. Applied to a Deep Research system, these principles enable local compressors as small as 3B parameters to recover 99% of frontier-LM accuracy at 26% of API costs.