mlsanityData to 7 Oct 2026

The Remarkable Robustness of LLMs: Stages of Inference?

arXiv 2406.19384 ↗First posted 27 Jun 2024cs.LG cs.AI cs.CLcited by 138

Not on the map. Why?

Abstract (arXiv)

We investigate the robustness of Large Language Models (LLMs) to structural interventions by deleting and swapping adjacent layers during inference. Surprisingly, models retain 72-95% of their original top-1 prediction accuracy without any fine-tuning. We find that performance degradation is not uniform across layers: interventions to the early and final layers cause the most degradation, while the model is remarkably robust to dropping middle layers. This pattern of localized sensitivity motivates our hypothesis of four stages of inference, observed across diverse model families and sizes: (1) detokenization, where local context is integrated to lift raw token embeddings into higher-level representations; (2) feature engineering, where task- and entity-specific features are iteratively refined; (3) prediction ensembling, where hidden states are aggregated into plausible next-token predictions; and (4) residual sharpening, where irrelevant features are suppressed to finalize the output distribution. Synthesizing behavioral and mechanistic evidence, we provide a framework for interpreting depth-dependent computations in LLMs.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Builds on 66 of its 83 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

66 of 83 references.

References · 66

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
38 more
  1. 2022In-context Learning and Induction Headscited by 930
  2. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  4. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  5. 2018Universal Transformerscited by 613
  6. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2023Textbooks Are All You Needcited by 561
  9. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  10. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  11. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  13. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  14. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  15. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  16. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  17. 2023Language Models Represent Space and Timecited by 306
  18. 2023In-Context Learning Creates Task Vectorscited by 302
  19. 2023Function Vectors in Large Language Modelscited by 292
  20. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  21. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  22. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  23. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  24. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  25. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  26. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  27. 2023An Overview of Catastrophic AI Riskscited by 214
  28. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  29. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  30. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  31. 2019Adaptively Sparse Transformerscited by 154
  32. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  33. 2022Analyzing Transformers in Embedding Spacecited by 131
  34. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  35. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  36. 2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
  37. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  38. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 53; 2026: 67.

138 papers.

Sort:

By area:

  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  2. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  3. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  4. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  5. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  6. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  7. 2025The Curse of Depth in Large Language Modelscited by 58
  8. 2025SuperBPE: Space Travel for Language Modelscited by 58
  9. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 57
  10. 2024Transformer Layers as Painterscited by 50
  11. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  12. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
38 more
  1. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  2. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  3. 2024Decomposing The Dark Matter of Sparse Autoencoderscited by 42
  4. 2024Efficient Dictionary Learning with Switch Sparse Autoencoderscited by 38
  5. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  6. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Modelscited by 31
  7. 2024Representational Analysis of Binding in Language Modelscited by 27
  8. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 27
  9. 2025The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representationscited by 26
  10. 2024Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layerscited by 25
  11. 2025Route Sparse Autoencoder to Interpret Large Language Modelscited by 25
  12. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  13. 2025The Dual-Route Model of Inductioncited by 22
  14. 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
  15. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13
  16. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  17. 2026Latent Introspection: Models Can Detect Prior Concept Injectionscited by 12
  18. 2026Training-Free Looped Transformerscited by 12
  19. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 12
  20. 2024Persistent Topological Features in Large Language Modelscited by 11
  21. 2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 11
  22. 2025Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Trainingcited by 9
  23. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 9
  24. 2025Learning to Skip the Middle Layers of Transformerscited by 8
  25. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 8
  26. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Modelscited by 7
  27. 2025Low-Rank Adapting Models for Sparse Autoencoderscited by 6
  28. 2025Leveraging the true depth of LLMscited by 6
  29. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
  30. 2025Interpreting Large Text-to-Image Diffusion Models with Dictionary Learningcited by 6
  31. 2025Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Modelscited by 6
  32. 2025Line of Sight: On Linear Representations in VLLMscited by 6
  33. 2025Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretrainingcited by 6
  34. 2026Anatomy of the Modality Gap: Dissecting the Internal States of End-to-End Speech LLMscited by 6
  35. 2024OMG-HD: A High-Resolution AI Weather Model for End-to-End Forecasts from Observationscited by 5
  36. 2025Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inferencecited by 5
  37. 2025Jailbreak Transferability Emerges from Shared Representationscited by 5
  38. 2025Punctuation and Predicates in Language Modelscited by 5

50 most cited of 138 All 138 (JSON)