# A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models

arXiv: https://arxiv.org/abs/2605.09515 · HTML: https://mlsanity.com/paper/2605.09515/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [BlockPruner: Fine-grained Pruning for Large Language Models](https://mlsanity.com/paper/2406.10594/) (2024)
- [Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning](https://mlsanity.com/paper/2601.06787/) (2026)
- [Natural Language Syntax Complies with the Free-Energy Principle](https://mlsanity.com/paper/2210.15098/) (2022)
- [Distributionally Robust Free Energy Principle for Decision-Making](https://mlsanity.com/paper/2503.13223/) (2025)
- [Structured Pruning for Diverse Best-of-N Reasoning Optimization](https://mlsanity.com/paper/2506.03978/) (2025)
- [CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning](https://mlsanity.com/paper/2411.17426/) (2024)
- [Beyond Shapley Values: Cooperative Games for the Interpretation of Machine Learning Models](https://mlsanity.com/paper/2506.13900/) (2025)
- [Active inference and artificial reasoning](https://mlsanity.com/paper/2512.21129/) (2025)
- [TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors](https://mlsanity.com/paper/2601.17958/) (2026)
- [D2Prune: Sparsifying Large Language Models via Dual Taylor Expansion and Attention Distribution Awareness](https://mlsanity.com/paper/2601.09176/) (2026)
- [Multi-Head Attention Is a Multi-Player Game](https://mlsanity.com/paper/2602.00861/) (2026)
- [NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics](https://mlsanity.com/paper/2603.18761/) (2026)
- [Bayesian Mechanics of Synaptic Learning under the Free Energy Principle](https://mlsanity.com/paper/2410.02972/) (2024)
- [Beyond Parallelism: Synergistic Computational Graph Effects in Multi-Head Attention](https://mlsanity.com/paper/2507.02944/) (2025)
- [SAP: Syntactic Attention Pruning for Transformer-based Language Models](https://mlsanity.com/paper/2512.19125/) (2025)
- [SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression](https://mlsanity.com/paper/2512.20635/) (2025)
- [A Collective Variational Principle Unifying Bayesian Inference, Game Theory, and Thermodynamics](https://mlsanity.com/paper/2604.27942/) (2026)

## Abstract (arXiv)

Large language models rely on multihead attention, but interactions among heads remain poorly understood. We apply the Game Theoretic Free Energy Principle (GTFEP): a framework casting multiagent systems as distributed variational inference to analyze attention heads as bounded rational agents. According to GTFEP, each head minimizes its variational free energy, and collective behavior follows a Gibbs distribution over coalition structures whose energy is decomposed into Harsanyi dividends. Using a tractable approximation (uniform prior, deterministic dynamics), coalition free energy reduces to joint Shannon entropy of discretized head outputs (argmax key index). Pairwise dividends become mutual information (nonnegative), while triple dividends correspond to interaction information and can be negative. On BERT, GPT2, and Llama with GSM8K, triple dividends are consistently negative, revealing higher order redundancy. The Nash FEP correspondence guarantees that stationary points of collective free energy are epsilon Nash equilibria; thus, heads with negligible contribution can be pruned with minimal performance loss. Pruning heads with low marginal contribution reduces computational cost with minimal performance loss: for example, pruning 20% of heads in GPT2 reduces FLOPs by 18%, increases throughput by 22%, and raises perplexity only modestly (from 28.4 to 33.4 on GSM8K). Our work shows GTFEP provides a principled foundation for analyzing and optimizing transformer architectures.