# Spectral Insights into Data-Oblivious Critical Layers in Large Language Models

arXiv: https://arxiv.org/abs/2506.00382 · HTML: https://mlsanity.com/paper/2506.00382/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 31 May 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives](https://mlsanity.com/paper/1909.01380/) (2019)
- [Safety Layers in Aligned Large Language Models: The Key to LLM Security](https://mlsanity.com/paper/2408.17003/) (2024)
- [LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning](https://mlsanity.com/paper/2403.17919/) (2024)
- [Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing](https://mlsanity.com/paper/2405.18166/) (2024)
- [BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models](https://mlsanity.com/paper/2406.17092/) (2024)
- [Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics](https://mlsanity.com/paper/2403.01509/) (2024)
- [Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers](https://mlsanity.com/paper/2109.08406/) (2021)
- [What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective](https://mlsanity.com/paper/2410.23743/) (2024)
- [Backdoor Federated Learning by Poisoning Backdoor-Critical Layers](https://mlsanity.com/paper/2308.04466/) (2023)
- [Understanding the Inner Workings of Language Models Through Representation Dissimilarity](https://mlsanity.com/paper/2310.14993/) (2023)
- [Tending Towards Stability: Convergence Challenges in Small Language Models](https://mlsanity.com/paper/2410.11451/) (2024)
- [Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injection](https://mlsanity.com/paper/2410.02330/) (2024)
- [Large Language Models for Predictive Analysis: How Far Are They?](https://mlsanity.com/paper/2505.17149/) (2025)

## Built on by (most cited first, 4 of 4)

- [From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures](https://mlsanity.com/paper/2511.22150/) (2025)
- [Representational Similarity and Model Behavior in Multi-Agent Interaction](https://mlsanity.com/paper/2606.07818/) (2026)
- [Do Reasoning Models Enhance Embedding Models?](https://mlsanity.com/paper/2601.21192/) (2026)
- [PLUME: Parameter-Efficient Personalization of Large Language Models via Low-Rank User Modulation in Shared Subspaces](https://mlsanity.com/paper/2609.04715/) (2026)

## Abstract (arXiv)

Understanding how feature representations evolve across layers in large language models (LLMs) is key to improving their interpretability and robustness. While recent studies have identified critical layers linked to specific functions or behaviors, these efforts typically rely on data-dependent analyses of fine-tuned models, limiting their use to post-hoc settings. In contrast, we introduce a data-oblivious approach to identify intrinsic critical layers in pre-fine-tuned LLMs by analyzing representation dynamics via Centered Kernel Alignment(CKA). We show that layers with significant shifts in representation space are also those most affected during fine-tuning--a pattern that holds consistently across tasks for a given model. Our spectral analysis further reveals that these shifts are driven by changes in the top principal components, which encode semantic transitions from rationales to conclusions. We further apply these findings to two practical scenarios: efficient domain adaptation, where fine-tuning critical layers leads to greater loss reduction compared to non-critical layers; and backdoor defense, where freezing them reduces attack success rates by up to 40%.