# Towards Effective Theory of LLMs: A Representation Learning Approach

arXiv: https://arxiv.org/abs/2605.09294 · HTML: https://mlsanity.com/paper/2605.09294/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 63 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 18 of 18)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024)
- [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024)
- [Priors in Time: Missing Inductive Biases for Language Model Interpretability](https://mlsanity.com/paper/2511.01836/) (2025)
- [TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2510.00739/) (2025)
- [Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models](https://mlsanity.com/paper/2507.12428/) (2025)
- [Prototype-Based Dynamic Steering for Large Language Models](https://mlsanity.com/paper/2510.05498/) (2025)
- [Value-Conflict Diagnostics Reveal Widespread Alignment Faking in Language Models](https://mlsanity.com/paper/2604.20995/) (2026)
- [The Origins of Computational Mechanics: A Brief Intellectual History and Several Clarifications](https://mlsanity.com/paper/1710.06832/) (2017)

## Abstract (arXiv)

We propose Representational Effective Theory (RET), a framework for describing large language model computation in terms of learned macrostates rather than microscopic details. RET learns these macrostates from hidden-state trajectories using a BYOL/JEPA-style self-supervised objective, coarse-graining activations into macrovariables that preserve higher-level structure relevant for prediction and interpretation. We evaluate whether these macrovariables are practically relevant for interpretability: RET yields temporally consistent states that reveal ``mental-state'' trajectories of reasoning, capture high-level semantic structure, support early prediction of behavioral outcomes such as sycophancy and alignment faking, while providing causal handles for steering generations toward interpretable computational phases. Together, these results suggest that LLM computation admits useful effective descriptions via RET: high-level, dynamically meaningful variables for interpretation, prediction, and control.