mlsanityData to 7 Oct 2026

Neural Sculpting: Uncovering hierarchically modular task structure in neural networks through pruning and network analysis

arXiv 2305.18402 ↗First posted 28 May 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Natural target functions and tasks typically exhibit hierarchical modularity -- they can be broken down into simpler sub-functions that are organized in a hierarchy. Such sub-functions have two important features: they have a distinct set of inputs (input-separability) and they are reused as inputs higher in the hierarchy (reusability). Previous studies have established that hierarchically modular neural networks, which are inherently sparse, offer benefits such as learning efficiency, generalization, multi-task learning, and transfer. However, identifying the underlying sub-functions and their hierarchical structure for a given task can be challenging. The high-level question in this work is: if we learn a task using a sufficiently deep neural network, how can we uncover the underlying hierarchy of sub-functions in that task? As a starting point, we examine the domain of Boolean functions, where it is easier to determine whether a task is hierarchically modular. We propose an approach based on iterative unit and edge pruning (during training), combined with network analysis for module detection and hierarchy inference. Finally, we demonstrate that this method can uncover the hierarchical modularity of a wide range of Boolean functions and two vision tasks based on the MNIST digits dataset.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 58 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

31 of 58 references.

References · 31

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2016Learning without Forgettingcited by 2,465
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  8. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  9. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  10. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  11. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  12. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
19 more
  1. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
  2. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  3. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  4. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  5. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 87
  6. 2020Lookahead: A Far-Sighted Alternative of Magnitude-based Pruningcited by 61
  7. 2021Continual Learning via Local Module Compositioncited by 61
  8. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  9. 2019The Generalization-Stability Tradeoff In Neural Network Pruningcited by 54
  10. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  11. 2020Lifelong Learning of Compositional Structurescited by 34
  12. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
  13. 2021Clusterability in Neural Networkscited by 23
  14. 2020PHEW: Constructing Sparse Networks that Learn Fast and Generalize Well without Training Datacited by 17
  15. 2017Modular Representation of Layered Neural Networkscited by 12
  16. 2018Interpreting Layered Neural Networks via Hierarchical Modular Representationcited by 11
  17. 2022Clustering units in neural networks: upstream vs downstream informationcited by 11
  18. 2015The evolutionary origins of hierarchycited by 9
  19. 2018Limits on representing Boolean functions by linear combinations of simple functions: thresholds, ReLUs, and low-degree polynomialscited by 6

Built on by

Not cited yet.