# Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory

arXiv: https://arxiv.org/abs/2501.00999 · HTML: https://mlsanity.com/paper/2501.00999/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2025.

## Built on by (most cited first, 9 of 9)

- [Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent](https://mlsanity.com/paper/2502.11740/) (2025)
- [HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs](https://mlsanity.com/paper/2507.17394/) (2025)
- [Topic Identification in LLM Input-Output Pairs through the Lens of Information Bottleneck](https://mlsanity.com/paper/2509.03533/) (2025)
- [Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization](https://mlsanity.com/paper/2510.16022/) (2025)
- [Is my model "mind blurting"? Interpreting the dynamics of reasoning tokens with Recurrence Quantification Analysis (RQA)](https://mlsanity.com/paper/2602.06266/) (2026)
- [WaterVIB: Learning Minimal Sufficient Watermark Representations via Variational Information Bottleneck](https://mlsanity.com/paper/2602.21508/) (2026)
- [FlowNIB: An Information Bottleneck Analysis of Bidirectional vs. Unidirectional Language Models](https://mlsanity.com/paper/2506.00859/) (2025)
- [Limits of Reliability and Scaling in Language Models](https://mlsanity.com/paper/2607.14112/) (2026)
- [When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors](https://mlsanity.com/paper/2610.13910/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks by understanding input information and predicting corresponding outputs. However, the internal mechanisms by which LLMs comprehend input and make effective predictions remain poorly understood. In this paper, we explore the working mechanism of LLMs in information processing from the perspective of Information Bottleneck Theory. We propose a non-training construction strategy to define a task space and identify the following key findings: (1) LLMs compress input information into specific task spaces (e.g., sentiment space, topic space) to facilitate task understanding; (2) they then extract and utilize relevant information from the task space at critical moments to generate accurate predictions. Based on these insights, we introduce two novel approaches: an Information Compression-based Context Learning (IC-ICL) and a Task-Space-guided Fine-Tuning (TS-FT). IC-ICL enhances reasoning performance and inference efficiency by compressing retrieved example information into the task space. TS-FT employs a space-guided loss to fine-tune LLMs, encouraging the learning of more effective compression and selection mechanisms. Experiments across multiple datasets validate the effectiveness of task space construction. Additionally, IC-ICL not only improves performance but also accelerates inference speed by over 40%, while TS-FT achieves superior results with a minimal strategy adjustment.