mlsanityData to 7 Oct 2026

Piecewise Linear Neural Networks and Deep Learning

arXiv 2206.09149 ↗First posted 18 Jun 2022cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

As a powerful modelling method, PieceWise Linear Neural Networks (PWLNNs) have proven successful in various fields, most recently in deep learning. To apply PWLNN methods, both the representation and the learning have long been studied. In 1977, the canonical representation pioneered the works of shallow PWLNNs learned by incremental designs, but the applications to large-scale data were prohibited. In 2010, the Rectified Linear Unit (ReLU) advocated the prevalence of PWLNNs in deep learning. Ever since, PWLNNs have been successfully applied to extensive tasks and achieved advantageous performances. In this Primer, we systematically introduce the methodology of PWLNNs by grouping the works into shallow and deep networks. Firstly, different PWLNN representation models are constructed with elaborated examples. With PWLNNs, the evolution of learning algorithms for data is presented and fundamental theoretical analysis follows up for in-depth understandings. Then, representative applications are introduced together with discussions and outlooks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 55 of its 212 references that are in the map. First posted on arXiv 18 Jun 2022.

Builds on

55 of 212 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2016Densely Connected Convolutional Networkscited by 6,691
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
38 more
  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  4. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  5. 2016Error bounds for approximations with deep ReLU networkscited by 567
  6. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  7. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  8. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  9. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  10. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  11. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  12. 2016On the Expressive Power of Deep Neural Networkscited by 302
  13. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  14. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  15. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  16. 2016Deep Learning without Poor Local Minimacited by 233
  17. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
  18. 2013Maxout Networkscited by 186
  19. 2019Complexity of Linear Regions in Deep Networkscited by 166
  20. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  21. 2016Identity Matters in Deep Learningcited by 107
  22. 2017Maximum Resilience of Artificial Neural Networkscited by 97
  23. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  24. 2019Deep Learning for Audio Signal Processingcited by 92
  25. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  26. 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 78
  27. 2017The loss surface of deep and wide neural networkscited by 74
  28. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 62
  29. 2019Certified Robustness for Top-k Predictions against Adversarial Perturbations via Randomized Smoothingcited by 54
  30. 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49
  31. 2021Partition-based formulations for mixed-integer optimization of trained ReLU neural networkscited by 39
  32. 2021Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?cited by 38
  33. 2016Convolutional Rectifier Networks as Generalized Tensor Decompositionscited by 32
  34. 2017Global optimality conditions for deep neural networkscited by 31
  35. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  36. 2015Deep Learning with S-shaped Rectified Linear Activation Unitscited by 26
  37. 2018G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Spacecited by 26
  38. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 3; 2026: 2.

7 papers.

  1. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  2. 2025Weaver: Kronecker Product Approximations of Spatiotemporal Attention for Traffic Network Forecastingcited by 1
  3. 2026ExLipBaB: Exact Lipschitz Constant Computation for Piecewise Linear Neural Networkscited by 1
  4. 2026AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networkscited by 1
  5. 2024Deep Learning Evidence for Global Optimality of Gerver's Sofacited by 0
  6. 2025Positivity sets of hinge functionscited by 0
  7. 2025Boundary-Aware Test-Time Adaptation for Zero-Shot Medical Image Segmentationcited by 0