mlsanityData to 14 Oct 2026

Decomposing the Depth Profile of Fine-Tuning

arXiv 2604.17177 ↗First posted 19 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Fine-tuning adapts pretrained networks to new objectives. Whether the resulting depth profile of representational change reflects an intrinsic property of the model or the magnitude of gradient flow has not been tested directly. We measure this profile across 240 fine-tuning runs spanning 15 models in four architecture families (encoder and decoder transformers, a state-space model, and an RNN) at scales from 125M to 6.9B parameters. Representational change concentrates in output-proximal layers in every standard-training run except one. We apply a per-layer control that equalizes |Delta W|/|W| across layers after each optimizer step. Under this control, the profile persists in some conditions and collapses in others. At 125M--350M, sequential-block architectures (BERT, OPT, GPT-2) retain the slope across tested objectives while parallel-block architectures (Pythia, CodeGen) retain it only for causal-language-modeling objectives. This architectural distinction narrows at 1.3B--1.4B, where both block types show positive equal-step slopes for CausalLM. Under standard training, profile shape is described by two additional axes: steepness tracks a training-free objective distance at initialization, and profile width is dominated by architecture. We treat the locality gradient, the depthwise slope of representational change, as a composite phenomenon whose components are scale-dependent.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 38 references that are in the map. First posted on arXiv 19 Apr 2026.

Builds on

34 of 38 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  12. 2014How transferable are features in deep neural networks?cited by 1,225
22 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  2. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  4. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  5. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  8. 2017Large Batch Training of Convolutional Networkscited by 535
  9. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  10. 2020What is being transferred in transfer learning?cited by 368
  11. 2024LoRA Learns Less and Forgets Lesscited by 328
  12. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  13. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  14. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  15. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  16. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  17. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  18. 2021Generalized Shape Metrics on Neural Representationscited by 112
  19. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  20. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 87
  21. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  22. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Mechanistic Investigation of Supervised Fine Tuning
  2. 2026Decoupling Internal Representational Changes and Causal Importance in Fine-Tuned Large Language Models
  3. 2025Layer-Wise Evolution of Representations in Fine-Tuned Transformers: Insights from Sparse AutoEncoders
  4. 2025Understanding Post-Training Structural Changes in Large Language Models
  5. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  6. 2026Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models
  7. 2026Emergence of Context Characteristics Sensitivity in Large Language Models
  8. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks