mlsanityData to 14 Oct 2026

Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models

arXiv 2305.17446 ↗First posted 27 May 2023cs.CL cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLMs) are known to be overly parameterized and have significant redundancy, indicating a small degree of freedom of the PLMs. Motivated by the observation, in this paper, we study the problem of re-parameterizing and fine-tuning PLMs from a new perspective: Discovery of intrinsic task-specific subspace. Specifically, by exploiting the dynamics of the fine-tuning process for a given task, the parameter optimization trajectory is learned to uncover its intrinsic task-specific subspace. A key finding is that PLMs can be effectively fine-tuned in the subspace with a small number of free parameters. Beyond, we observe some outlier dimensions emerging during fine-tuning in the subspace. Disabling these dimensions degrades the model performance significantly. This suggests that these dimensions are crucial to induce task-specific knowledge to downstream tasks.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

28 of 49 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  11. 2018Neural Network Acceptability Judgmentscited by 958
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
16 more
  1. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  2. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  3. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  4. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  5. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  6. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  7. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 156
  8. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  9. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  10. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  11. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  12. 2021Subspace Adversarial Trainingcited by 41
  13. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 35
  14. 2020Improving Neural Network Training in Low Dimensional Random Basescited by 19
  15. 2022No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Modelscited by 15
  16. 2022Few-Shot Learning by Dimensionality Reduction in Gradient Spacecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 6.

19 papers.

Sort:

By area:

  1. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  2. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  3. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 34
  4. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  5. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  6. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 26
  7. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 20
  8. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 16
  9. 2025Learning to Interpret Weight Differences in Language Modelscited by 11
  10. 2025Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behaviorcited by 8
  11. 2025Uni-LoRA: One Vector is All You Needcited by 7
  12. 2025DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modelingcited by 5
7 more
  1. 2024Unlocking Efficiency: Adaptive Masking for Gene Transformer Modelscited by 2
  2. 2025Recover-to-Forget: Gradient Reconstruction from LoRA for Efficient LLM Unlearningcited by 1
  3. 2026Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptationcited by 1
  4. 2026The Practicality of Normalizing Flow Test-Time Training in Bayesian Inference for Agent-Based Modelscited by 0
  5. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0
  6. 2026SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspacescited by 0
  7. 2026SandwichQuant: Which Parameters Matter Before and After Quantization?cited by 0