mlsanityData to 14 Oct 2026

Deep ReLU Networks Have Surprisingly Few Activation Patterns

arXiv 1906.00904 ↗First posted 3 Jun 2019stat.ML cs.LG math.STcited by 161

Not on the map. Why?

Abstract (arXiv)

The success of deep networks has been attributed in part to their expressivity: per parameter, deep networks can approximate a richer class of functions than shallow networks. In ReLU networks, the number of activation patterns is one measure of expressivity; and the maximum number of patterns grows exponentially with the depth. However, recent work has showed that the practical expressivity of deep networks - the functions they can learn rather than express - is often far from the theoretical maximum. In this paper, we show that the average number of activation patterns for ReLU networks at initialization is bounded by the total number of neurons raised to the input dimension. We show empirically that this bound, which is independent of the depth, is tight both at initialization and during training, even on memorization tasks that should maximize the number of activation patterns. Our work suggests that realizing the full expressivity of deep networks may not be possible in practice, at least with current methods.

Key facts. Cited by 161 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 29; 2022: 31; 2023: 30; 2024: 28; 2025: 23; 2026: 20.

161 papers.

Sort:

By area:

  1. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
  2. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  3. 2022Neural Architecture Search for Spiking Neural Networkscited by 53
  4. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  5. 2021Provable Repair of Deep Neural Networkscited by 42
  6. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  7. 2024Where Do We Stand with Implicit Neural Representations? A Technical and Performance Surveycited by 42
  8. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  9. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  10. 2023Hidden symmetries of ReLU networkscited by 39
  11. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  12. 2021Consistent Counterfactuals for Deep Modelscited by 37
38 more
  1. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  2. 2022Interpreting Neural Networks through the Polytope Lenscited by 33
  3. 2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 28
  4. 2023Understanding the Feature Norm for Out-of-Distribution Detectioncited by 27
  5. 2022Improved Bounds on Neural Complexity for Representing Piecewise Linear Functionscited by 26
  6. 2022DCI-ES: An Extended Disentanglement Framework with Connections to Identifiabilitycited by 23
  7. 2022Algorithmic Determination of the Combinatorial Structure of the Linear Regions of ReLU Neural Networkscited by 22
  8. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  9. 2022Functional dimension of feedforward ReLU neural networkscited by 21
  10. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  11. 2021Scaling Up Exact Neural Network Compression by ReLU Stabilitycited by 19
  12. 2021Depth separation beyond radial functionscited by 18
  13. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  14. 2021On the Expected Complexity of Maxout Networkscited by 15
  15. 2022Detection of out-of-distribution samples using binary neuron activation patternscited by 15
  16. 2023Polyhedral Complex Extraction from ReLU Networks using Edge Subdivisioncited by 14
  17. 2021Reliably-stabilizing piecewise-affine neural network controllerscited by 13
  18. 2021Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verificationcited by 13
  19. 2022Are All Linear Regions Created Equal?cited by 13
  20. 2024FairProof : Confidential and Certifiable Fairness for Neural Networkscited by 13
  21. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  22. 2021A Simple and Efficient Sampling-based Algorithm for General Reachability Analysiscited by 12
  23. 2022Towards Reliable Neural Specificationscited by 12
  24. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  25. 2022Understanding Deep Neural Function Approximation in Reinforcement Learning via ε-Greedy Explorationcited by 11
  26. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  27. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  28. 2023ReLU Neural Networks, Polyhedral Decompositions, and Persistent Homologcited by 10
  29. 2024Optimization Over Trained Neural Networks: Taking a Relaxing Walkcited by 10
  30. 2021Test Sample Accuracy Scales with Training Sample Density in Neural Networkscited by 9
  31. 2022Origami in N dimensions: How feed-forward networks manufacture linear separabilitycited by 9
  32. 2022Lower and Upper Bounds for Numbers of Linear Regions of Graph Convolutional Networkscited by 9
  33. 2024Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimallycited by 9
  34. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  35. 2022Local and global topological complexity measures OF ReLU neural network functionscited by 8
  36. 2022Curved Representation Space of Vision Transformerscited by 8
  37. 2023Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regionscited by 8
  38. 2024Intriguing Equivalence Structures of the Embedding Space of Vision Transformerscited by 8

50 most cited of 161 All 161 (JSON)