mlsanityData to 7 Oct 2026

Functional Degeneracy in Neural Networks: Measurement and Pruning

arXiv 2608.30741 ↗First posted 31 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, defined as the number of leading behavioral-Hessian eigendirections required to recover a trained model's performance. Using the behavioral recovery rank as a geometric benchmark for compression, we find that structural and magnitude pruning retain more degrees of freedom, even after the task is saturated. This gap suggests that functional redundancy is distributed across parameter directions and is not exposed by individual weights or neurons.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

26 of 39 references.

References · 26

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  9. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  10. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
14 more
  1. 2020What is the State of Neural Network Pruning?cited by 578
  2. 2019Importance Estimation for Neural Network Pruningcited by 531
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  4. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  5. 2021Explaining Neural Scaling Lawscited by 353
  6. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  7. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  8. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  9. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  10. 2023Hidden symmetries of ReLU networkscited by 38
  11. 2022Statistical exploration of the Manifold Hypothesiscited by 29
  12. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  13. 2022Functional dimension of feedforward ReLU neural networkscited by 20
  14. 2024Using Degeneracy in the Loss Landscape for Mechanistic Interpretabilitycited by 12

Built on by

Not cited yet.