mlsanityData to 7 Oct 2026

On the Expected Complexity of Maxout Networks

arXiv 2107.00379 ↗First posted 1 Jul 2021stat.ML cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Learning with neural networks relies on the complexity of the representable functions, but more importantly, the particular assignment of typical parameters to functions of different complexity. Taking the number of activation regions as a complexity measure, recent works have shown that the practical complexity of deep ReLU networks is often far from the theoretical maximum. In this work, we show that this phenomenon also occurs in networks with maxout (multi-argument) activation functions and when considering the decision boundaries in classification tasks. We also show that the parameter space has a multitude of full-dimensional regions with widely different complexity, and obtain nontrivial lower bounds on the expected complexity. Finally, we investigate different parameter initialization procedures and show that they can increase the speed of convergence in training.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 30 of its 67 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

30 of 67 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  5. 2016On the Expressive Power of Deep Neural Networkscited by 302
  6. 2016Benefits of depth in neural networkscited by 248
  7. 2021Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networkscited by 219
  8. 2013Maxout Networkscited by 186
  9. 2019Complexity of Linear Regions in Deep Networkscited by 166
  10. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  11. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  12. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
18 more
  1. 2018Tropical Geometry of Deep Neural Networkscited by 92
  2. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  3. 2015Representation Benefits of Deep Feedforward Networkscited by 84
  4. 2019The Geometry of Deep Networks: Power Diagram Subdivisioncited by 53
  5. 2018Provable Robustness of ReLU networks via Maximization of Linear Regionscited by 51
  6. 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49
  7. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  8. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  9. 2020Empirical Studies on the Properties of Linear Regions in Deep Neural Networkscited by 36
  10. 2020Lossless Compression of Deep Neural Networkscited by 34
  11. 2018A Tropical Approach to Neural Networks with Piecewise Linear Activationscited by 26
  12. 2018Empirical Bounds on Linear Regions of Deep Rectifier Networkscited by 23
  13. 2019Towards Robust, Locally Linear Deep Networkscited by 20
  14. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  15. 2018A Framework for the construction of upper bounds on the number of affine linear regions of ReLU feed-forward neural networkscited by 15
  16. 2021Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networkscited by 7
  17. 2019A Sober Look at Neural Network Initializationscited by 4
  18. 2020A General Computational Framework to Measure the Expressiveness of Complex Networks Using a Tighter Upper Bound of Linear Regionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 1; 2025: 3; 2026: 5.

15 papers.

Sort:

  1. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  2. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  3. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  4. 2022Lower and Upper Bounds for Numbers of Linear Regions of Graph Convolutional Networkscited by 9
  5. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  6. 2023Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regionscited by 8
  7. 2025The Computational Complexity of Counting Linear Regions in ReLU Neural Networkscited by 7
  8. 2025Constraining the outputs of ReLU neural networkscited by 3
  9. 2023Expected Gradients of Maxout Networks and Consequences to Parameter Initializationcited by 1
  10. 2026Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networkscited by 1
  11. 2026AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networkscited by 1
  12. 2026Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networkscited by 1
3 more
  1. 2025Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogatecited by 0
  2. 2026A Rigorous, Tractable Measure of Model Complexitycited by 0
  3. 2026Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexitycited by 0