mlsanityData to 14 Oct 2026

Measuring the Intrinsic Dimension of Objective Landscapes

arXiv 1804.08838 ↗First posted 24 Apr 2018cs.LG cs.NE stat.MLcited by 370

Not on the map. Why?

Abstract (arXiv)

Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem. But how accurate are such notions? How many parameters are really needed? In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace. We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape. The approach is simple to implement, computationally tractable, and produces several suggestive conclusions. Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes. This latter result has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold. Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10. In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution. A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.

Key facts. Cited by 370 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 33; 2022: 40; 2023: 51; 2024: 87; 2025: 65; 2026: 93.

370 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  5. 2021Robust fine-tuning of zero-shot modelscited by 797
  6. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  7. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  8. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  9. 2024LoRA Learns Less and Forgets Lesscited by 328
  10. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  11. 2022Matryoshka Representation Learningcited by 282
  12. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
38 more
  1. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  2. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  3. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  4. 2023Knowledge Editing for Large Language Models: A Surveycited by 194
  5. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  6. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 173
  7. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  8. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  9. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  10. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 112
  11. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  12. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 104
  13. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
  14. 2023Deep Model Fusion: A Surveycited by 91
  15. 2023Modular Deep Learningcited by 83
  16. 2021Learning Neural Network Subspacescited by 82
  17. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  18. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  19. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  20. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 77
  21. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 76
  22. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  23. 2023Building Emotional Support Chatbots in the Era of LLMscited by 62
  24. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  25. 2022Transferability in Deep Learning: A Surveycited by 57
  26. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  27. 2025Deep Learning is Not So Mysterious or Differentcited by 56
  28. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 55
  29. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  30. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  31. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  32. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  33. 2023Transformers learn through gradual rank increasecited by 52
  34. 2022Generalizing to New Physical Systems via Context-Informed Dynamics Modelcited by 50
  35. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  36. 2024Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structurescited by 50
  37. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  38. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44

50 most cited of 370 All 370 (JSON)