mlsanityData to 7 Oct 2026

Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities

arXiv 2602.00693 ↗First posted 31 Jan 2026cs.LG math.AG math.ATcited by 6

Not on the map. Why?

Abstract (arXiv)

Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 49 of its 76 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

49 of 76 references.

References · 49

  1. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  8. 2018Rethinking the Value of Network Pruningcited by 595
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  11. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  12. 2017Sharp Minima Can Generalize For Deep Netscited by 462
37 more
  1. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  2. 2023DepGraph: Towards Any Structural Pruningcited by 202
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  4. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  5. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  6. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  7. 2020Dynamic Model Pruning with Feedbackcited by 125
  8. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  9. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  10. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
  11. 2019Winning the Lottery with Continuous Sparsificationcited by 99
  12. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  13. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  14. 2020Operation-Aware Soft Channel Pruning using Differentiable Maskscited by 74
  15. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  16. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  17. 2020Deep Learning is Singular, and That's Goodcited by 49
  18. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  19. 2023Hidden symmetries of ReLU networkscited by 38
  20. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  21. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  22. 2025Symmetry in Neural Network Parameter Spacescited by 31
  23. 2021SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruningcited by 30
  24. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  25. 2020The Representation Theory of Neural Networkscited by 18
  26. 2024Geometry of Lightning Self-Attention: Identifiability and Dimensioncited by 16
  27. 2016Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methodscited by 15
  28. 2019Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximationcited by 14
  29. 2016DropNeuron: Simplifying the Structure of Deep Neural Networkscited by 11
  30. 2024Algebraic Complexity and Neurovariety of Linear Convolutional Networkscited by 10
  31. 2025Geometry and Optimization of Shallow Polynomial Networkscited by 10
  32. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  33. 2024Estimating the Local Learning Coefficient at Scalecited by 6
  34. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
  35. 2022Analysis of function approximation and stability of general DNNs in directed acyclic graphs using un-rectifying analysiscited by 3
  36. 20224Ward: a Relayering Strategy for Efficient Training of Arbitrarily Complex Directed Acyclic Graphscited by 3
  37. 2023Beyond Multilayer Perceptrons: Investigating Complex Topologies in Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Most ReLU Networks Admit Identifiable Parameterscited by 5
  2. 2026The Symmetries of Three-Layer ReLU Networkscited by 2
  3. 2026From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networkscited by 0
  4. 2026Stratifying Reinforcement Learning with Signal Temporal Logiccited by 0
  5. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  6. 2026On Parameter Symmetries and Conservation Laws in Gradient Flowcited by 0