# Does Representation Matter? Exploring Intermediate Layers in Large Language Models

arXiv: https://arxiv.org/abs/2412.09563 · HTML: https://mlsanity.com/paper/2412.09563/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank](https://mlsanity.com/paper/2210.02885/) (2022)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Not All Layers of LLMs Are Necessary During Inference](https://mlsanity.com/paper/2403.02181/) (2024)
- [BM25S: Orders of magnitude faster lexical search via eager sparse scoring](https://mlsanity.com/paper/2407.03618/) (2024)
- [To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review](https://mlsanity.com/paper/2304.09355/) (2023)
- [Eliciting Latent Knowledge from Quirky Language Models](https://mlsanity.com/paper/2312.01037/) (2023)
- [Emergence of Separable Manifolds in Deep Language Representations](https://mlsanity.com/paper/2006.01095/) (2020)
- [Information Theory with Kernel Methods](https://mlsanity.com/paper/2202.08545/) (2022)
- [Reverse Engineering Self-Supervised Learning](https://mlsanity.com/paper/2305.15614/) (2023)
- [Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language](https://mlsanity.com/paper/2311.04930/) (2023)
- [LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures](https://mlsanity.com/paper/2312.04000/) (2023)
- [Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning](https://mlsanity.com/paper/2206.13378/) (2022)
- [Information Flow in Deep Neural Networks](https://mlsanity.com/paper/2202.06749/) (2022)
- [DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropies](https://mlsanity.com/paper/2301.08164/) (2023)

## Built on by (most cited first, 30 of 34)

- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [A Survey on Latent Reasoning](https://mlsanity.com/paper/2507.06203/) (2025)
- [Defending against Indirect Prompt Injection by Instruction Detection](https://mlsanity.com/paper/2505.06311/) (2025)
- [BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Models](https://mlsanity.com/paper/2505.18132/) (2025)
- [The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models](https://mlsanity.com/paper/2501.10573/) (2025)
- [Advancing LLM Safe Alignment with Safety Representation Ranking](https://mlsanity.com/paper/2505.15710/) (2025)
- [ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer](https://mlsanity.com/paper/2501.15570/) (2025)
- [HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs](https://mlsanity.com/paper/2507.17394/) (2025)
- [Efficient numeracy in language models through single-token number embeddings](https://mlsanity.com/paper/2510.06824/) (2025)
- [Entropy-Lens: Uncovering Decision Strategies in LLMs](https://mlsanity.com/paper/2502.16570/) (2025)
- [Lightweight Safety Classification Using Pruned Language Models](https://mlsanity.com/paper/2412.13435/) (2024)
- [Improving Visual Reasoning with Iterative Evidence Refinement](https://mlsanity.com/paper/2603.14117/) (2026)
- [Explanations of Large Language Models Explain Language Representations in the Brain](https://mlsanity.com/paper/2502.14671/) (2025)
- [Where Did It Go Wrong? Attributing Undesirable LLM Behaviors via Representation Gradient Tracing](https://mlsanity.com/paper/2510.02334/) (2025)
- [Latent Structure of Affective Representations in Large Language Models](https://mlsanity.com/paper/2604.07382/) (2026)
- [CHAIR -- Classifier of Hallucination as Improver](https://mlsanity.com/paper/2501.02518/) (2025)
- [Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoning](https://mlsanity.com/paper/2508.10057/) (2025)
- [Language Ranker: A Lightweight Ranking framework for LLM Decoding](https://mlsanity.com/paper/2510.21883/) (2025)
- [GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models](https://mlsanity.com/paper/2512.15560/) (2025)
- [Decomposing how prompting steers behavior](https://mlsanity.com/paper/2606.03093/) (2026)
- [Intra-request branch orchestration for efficient LLM reasoning](https://mlsanity.com/paper/2509.24957/) (2025)
- [LLaMAX2: Your Translation-Enhanced Model also Performs Well in Reasoning](https://mlsanity.com/paper/2510.09189/) (2025)
- [Concept Attractors in LLMs and their Applications](https://mlsanity.com/paper/2601.11575/) (2025)
- [Language-Pretraining-Induced Bias: A Strong Foundation for General Vision Tasks](https://mlsanity.com/paper/2604.01833/) (2026)
- [Geometry of Human Perceptual Domains Emerges Transiently in LLM Representations](https://mlsanity.com/paper/2605.27970/) (2026)
- [PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatorics](https://mlsanity.com/paper/2609.25438/) (2026)
- [Blind Spot Navigation: Evolutionary Discovery of Sensitive Semantic Concepts for LVLMs](https://mlsanity.com/paper/2505.15265/) (2025)
- [Multilingual Embedding Probes Fail to Generalize Across Learner Corpora](https://mlsanity.com/paper/2604.07095/) (2026)
- [Towards Understanding the Robustness of Sparse Autoencoders](https://mlsanity.com/paper/2604.18756/) (2026)
- [Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models](https://mlsanity.com/paper/2606.19750/) (2026)

## Abstract (arXiv)

Understanding what defines a good representation in large language models (LLMs) is fundamental to both theoretical understanding and practical applications. In this paper, we investigate the quality of intermediate representations in various LLM architectures, including Transformers and State Space Models (SSMs). We find that intermediate layers often yield more informative representations for downstream tasks than the final layers. To measure the representation quality, we adapt and apply a suite of metrics - such as prompt entropy, curvature, and augmentation-invariance - originally proposed in other contexts. Our empirical study reveals significant architectural differences, how representations evolve throughout training, and how factors like input randomness and prompt length affect each layer. Notably, we observe a bimodal pattern in the entropy of some intermediate layers and consider potential explanations tied to training data. Overall, our results illuminate the internal mechanics of LLMs and guide strategies for architectural optimization and training.