# Interpreting Reasoning of Large Language Models via Partial Information Decomposition

arXiv: https://arxiv.org/abs/2610.00571 · HTML: https://mlsanity.com/paper/2610.00571/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs](https://mlsanity.com/paper/2406.15927/) (2024)
- [PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models](https://mlsanity.com/paper/2501.03124/) (2025)
- [Understanding Chain-of-Thought in LLMs through Information Theory](https://mlsanity.com/paper/2411.11984/) (2024)
- [Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoning](https://mlsanity.com/paper/2501.15602/) (2025)
- [Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens](https://mlsanity.com/paper/2505.18237/) (2025)
- [A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition](https://mlsanity.com/paper/2105.04187/) (2021)
- [Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications](https://mlsanity.com/paper/2306.04539/) (2023)
- [Gaussian Partial Information Decomposition: Bias Correction and Application to High-dimensional Data](https://mlsanity.com/paper/2307.10515/) (2023)
- [DiffusionPID: Interpreting Diffusion via Partial Information Decomposition](https://mlsanity.com/paper/2406.05191/) (2024)
- [Fairness Under Feature Exemptions: Counterfactual and Observational Measures](https://mlsanity.com/paper/2006.07986/) (2020)
- [Quantifying Knowledge Distillation Using Partial Information Decomposition](https://mlsanity.com/paper/2411.07483/) (2024)
- [More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learning](https://mlsanity.com/paper/2307.00651/) (2023)
- [Demystifying Local and Global Fairness Trade-offs in Federated Learning Using Partial Information Decomposition](https://mlsanity.com/paper/2307.11333/) (2023)
- [Towards Formalizing Spuriousness of Biased Datasets Using Partial Information Decomposition](https://mlsanity.com/paper/2407.00482/) (2024)
- [A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models](https://mlsanity.com/paper/2603.29676/) (2026)
- [A Unified View of Group Fairness Tradeoffs Using Partial Information Decomposition](https://mlsanity.com/paper/2406.04562/) (2024)
- [Learning Invariant Graph Representations Through Redundant Information](https://mlsanity.com/paper/2512.06154/) (2025)

## Abstract (arXiv)

Large reasoning models (LRMs) have achieved substantial improvements in solving complex mathematical problems, but often produce lengthy, repetitive, or erroneous reasoning trajectories. In this work, we introduce a new interpretability framework, SLIDER, to evaluate the quality of the reasoning process. SLIDER leverages an emerging body of work from information theory called Partial Information Decomposition to disentangle the information about the final answer between two consecutive reasoning steps into non-negative components: unique information (in preceding steps or current step), redundant information, and synergistic information. Building on this decomposition, we propose the *Step-wise Repetitive Reasoning Index (Step-RRI)*, a theoretically grounded measure that assesses whether the answer-relevant information in the current step Si is predominantly redundant with the past steps S<i, relative to its unique and synergistic contributions. To evaluate the effectiveness of Step-RRI in detecting repetitiveness, we apply SLIDER to the redundancy class of the PRMBench dataset where Step-RRI improves step-level redundancy identification accuracy by over 10 points compared to embedding-similarity and information-gain baselines. Next, we define *Trajectory-RRI*, an aggregate measure of repetitiveness for an individual reasoning trajectory. To demonstrate its practical relevance, we show that average Trajectory-RRI strongly correlates with actual reasoning length across QwQ-32B, DeepSeek-R1-Distill-Qwen-32B, and GPT-4.1, motivating its use as a signal for improving reasoning efficiency. Finally, we introduce *Trajectory-RRI-guided data selection for fine-tuning*, demonstrating that selecting training data based on Trajectory-RRI can improve a fine-tuned model's reasoning efficiency while largely preserving its task performance.