mlsanityData to 7 Oct 2026

Transformer Layers as Painters

arXiv 2407.09298 ↗First posted 12 Jul 2024cs.CLcited by 50

Not on the map. Why?

Abstract (arXiv)

Despite their nearly universal adoption for large language models, the internal workings of transformers are not well understood. We aim to better understand the impact of removing or reorganizing information throughout the layers of a pretrained transformer. Such an understanding could both yield better usage of existing models as well as to make architectural improvements to produce new variants. We present a series of empirical studies on frozen models that show that the lower and final layers of pretrained transformers differ from middle layers, but that middle layers have a surprising amount of uniformity. We further show that some classes of problems have robustness to skipping layers, running the layers in an order different from how they were trained, or running the layers in parallel. Our observations suggest that even frozen pretrained models may gracefully trade accuracy for latency by skipping layers or running layers in parallel.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 19 of its 31 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

19 of 31 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
7 more
  1. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  2. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  3. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  4. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  5. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 48
  6. 2022A Study on Transformer Configuration and Training Objectivecited by 4
  7. 2024VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 33; 2026: 13.

50 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  2. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  3. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  4. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  5. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  6. 2024Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancementcited by 21
  7. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practicescited by 21
  8. 2025BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Modelscited by 18
  9. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  10. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13
  11. 2024Looking Beyond The Top-1: Transformers Determine Top Tokens In Ordercited by 12
  12. 2025Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMscited by 10
38 more
  1. 2024Self-Data Distillation for Recovering Quality in Pruned Large Language Modelscited by 8
  2. 2025Identifying and Evaluating Inactive Heads in Pretrained LLMscited by 8
  3. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  4. 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Modelscited by 7
  5. 2025The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilitiescited by 7
  6. 2025Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinementcited by 5
  7. 2025Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysiscited by 5
  8. 2025Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Mergingcited by 4
  9. 2025LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgradescited by 4
  10. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 4
  11. 2025Auto-Compressing Networkscited by 4
  12. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognitioncited by 4
  13. 2026A Brain-like Synergistic Core in LLMs Drives Behaviour and Learningcited by 4
  14. 2026Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Modelscited by 4
  15. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  16. 2026Loop the Loopies!cited by 4
  17. 2025SplitFrozen: Split Learning with Device-side Model Frozen for Fine-Tuning LLM on Heterogeneous Resource-Constrained Devicescited by 3
  18. 2025SEReDeEP: Hallucination Detection in Retrieval-Augmented Models via Semantic Entropy and Context-Parameter Fusioncited by 3
  19. 2025Recurrence Meets Transformers for Universal Multimodal Retrievalcited by 3
  20. 2026Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Modelscited by 3
  21. 2026Investigating Single-Block Recurrence in Vision Transformers for Image Recognitioncited by 3
  22. 2025Hyper-SET: Designing Transformers via Hyperspherical Energy Minimizationcited by 2
  23. 2025LoKI: Low-damage Knowledge Implanting of Large Language Modelscited by 2
  24. 2025Zero-Shot Vision Encoder Grafting via LLM Surrogatescited by 2
  25. 2025Think Right, Not More: Test-Time Scaling for Numerical Claim Verificationcited by 2
  26. 2025Decoding Vision Transformers: the Diffusion Steering Lenscited by 1
  27. 2025A Review of Developmental Interpretability in Large Language Modelscited by 1
  28. 2025Towards Understanding Layer Contributions in Tabular In-Context Learning Modelscited by 1
  29. 2026Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervisioncited by 1
  30. 2026Temporal Recurrence Favors Fewer Layerscited by 1
  31. 2026Programs-of-Layers in LLMs through the Lens of Cortical Areascited by 1
  32. 2025Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignmentcited by 0
  33. 2025Direct Multi-Token Decodingcited by 0
  34. 2025Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneitycited by 0
  35. 2026FlattenGPT: Depth Compression for Transformer with Layer Flatteningcited by 0
  36. 2026Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Taskcited by 0
  37. 2026Post-Hoc Understanding of Metaphor Processing in Decoder-Only Language Models via Conditional Scale Entropycited by 0
  38. 2026Layer-Informed Fine-Tuning via Three-Stage Functional Segmentation of LLMscited by 0