mlsanityData to 7 Oct 2026

Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

arXiv 2012.13255 ↗First posted 22 Dec 2020cs.LG cs.CLcited by 658

Not on the map. Why?

Cited by 22 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well understood, especially in the low data regime. Why can we use relatively vanilla gradient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of parameters on datasets with only hundreds or thousands of labeled examples? In this paper, we argue that analyzing fine-tuning through the lens of intrinsic dimension provides us with empirical and theoretical intuitions to explain this remarkable phenomenon. We empirically show that common pre-trained models have a very low intrinsic dimension; in other words, there exists a low dimension reparameterization that is as effective for fine-tuning as the full parameter space. For example, by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter performance levels on MRPC. Furthermore, we empirically show that pre-training implicitly minimizes intrinsic dimension and, perhaps surprisingly, larger models tend to have lower intrinsic dimension after a fixed number of pre-training updates, at least in part explaining their extreme effectiveness. Lastly, we connect intrinsic dimensionality with low dimensional task representations and compression based generalization bounds to provide intrinsic-dimension-based generalization bounds that are independent of the full parameter count.

Key facts. Cited by 658 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 32; 2023: 101; 2024: 172; 2025: 159; 2026: 177.

658 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  8. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  9. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  10. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  11. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  12. 2024LoRA Learns Less and Forgets Lesscited by 313
38 more
  1. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  2. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  3. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  4. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  5. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  6. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  7. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  8. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  9. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  10. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  11. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  12. 2024How do Large Language Models Handle Multilingualism?cited by 148
  13. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  14. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  15. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  16. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  17. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  18. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 118
  19. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  20. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  21. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 111
  22. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  23. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  24. 2023The geometry of hidden representations of large transformer modelscited by 108
  25. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  26. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  27. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  28. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  29. 2023Modular Deep Learningcited by 82
  30. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  31. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  32. 2024Multi-LoRA Composition for Image Generationcited by 80
  33. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  34. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  35. 2024NetLLM: Adapting Large Language Models for Networkingcited by 76
  36. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
  37. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  38. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73

50 most cited of 658 All 658 (JSON)