mlsanityData to 14 Oct 2026

High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation

arXiv 2512.23410 ↗First posted 29 Dec 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

State-of-the-art models rely on massive widths despite exhibiting low Intrinsic Dimension (ID). We posit that this redundancy serves the non-convex optimization search rather than the final representation. We validate this hypothesis by decoupling the solution geometry via data-independent random projections, demonstrating that ResNet, ViT, and BERT representations can be compressed by up to 16x with negligible performance degradation of around 1%. Notably, these oblivious projections achieve parity with PCA and learned baselines, confirming the solution manifold is intrinsically robust. These findings establish the foundation for Subspace-Native Distillation: a paradigm where student models target this intrinsic manifold directly, bypassing the high-dimensional optimization bottleneck to realize the vision of "Train Big, Deploy Small"

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

22 of 33 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  10. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  11. 2019Contrastive Multiview Codingcited by 1,301
  12. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
10 more
  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  2. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  3. 2019On the Efficacy of Knowledge Distillationcited by 381
  4. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  5. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
  6. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  7. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  8. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  9. 2021Towards Understanding Knowledge Distillationcited by 158
  10. 2018The Singular Values of Convolutional Layerscited by 95

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning
  2. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry
  3. 2025Behavior and Representation in Open-Weight Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selection
  4. 2022Benign Autoencoders
  5. 2026SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits
  6. 2026Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation
  7. 2025What We Don't C: Manifold Disentanglement for Structured Discovery
  8. 2026Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability