mlsanityData to 14 Oct 2026

Mad Max: Affine Spline Insights into Deep Learning

arXiv 1805.06576 ↗First posted 17 May 2018stat.ML cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

We build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs), which provide a powerful portal through which to view and analyze their inner workings. For instance, conditioned on the input signal, the output of a MASO DN can be written as a simple affine transformation of the input. This implies that a DN constructs a set of signal-dependent, class-specific templates against which the signal is compared via a simple inner product; we explore the links to the classical theory of optimal classification via matched filters and the effects of data memorization. Going further, we propose a simple penalty term that can be added to the cost function of any DN learning algorithm to force the templates to be orthogonal with each other; this leads to significantly improved classification performance and reduced overfitting with no change to the DN architecture. The spline partition of the input signal space that is implicitly induced by a MASO directly links DNs to the theory of vector quantization (VQ) and K-means clustering, which opens up new geometric avenue to study how DNs organize signals in a hierarchical fashion. To validate the utility of the VQ interpretation, we develop and validate a new distance metric for signals and images that quantifies the difference between their VQ encodings. (This paper is a significantly expanded version of A Spline Theory of Deep Learning from ICML 2018.)

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 15; 2023: 9; 2024: 6; 2025: 7; 2026: 8.

55 papers.

Sort:

By area:

  1. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 120
  2. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  3. 2022Mechanistic Mode Connectivitycited by 50
  4. 2022ExSpliNet: An interpretable and expressive spline-based neural networkcited by 33
  5. 2022Interpreting Neural Networks through the Polytope Lenscited by 33
  6. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 31
  7. 2021MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retrainingcited by 23
  8. 2022Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Valuescited by 21
  9. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  10. 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
  11. 2022Neural Networks are Decision Treescited by 17
  12. 2023Polyhedral Complex Extraction from ReLU Networks using Edge Subdivisioncited by 14
38 more
  1. 2023Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generationcited by 12
  2. 2022Singular Value Perturbation and Deep Network Optimizationcited by 11
  3. 2022Origami in N dimensions: How feed-forward networks manufacture linear separabilitycited by 9
  4. 2024Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularizationcited by 9
  5. 2022Batch Normalization Explainedcited by 8
  6. 2025Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stabilitycited by 6
  7. 2022Decision boundaries and convex hulls in the feature space that deep learning functions learn from imagescited by 5
  8. 2022What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundariescited by 5
  9. 2023Deep ReLU Networks Have Surprisingly Simple Polytopescited by 5
  10. 2023SmoothHess: ReLU Network Feature Interactions via Stein's Lemmacited by 5
  11. 2021Fast Jacobian-Vector Product for Deep Networkscited by 4
  12. 2022Reverse Engineering ℓp attacks: A block-sparse optimization approach with recovery guaranteescited by 4
  13. 2021Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for Few-Shot Learningcited by 3
  14. 2024Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initializationcited by 3
  15. 2026The Linear Centroids Hypothesis: Features as Directions Learned by Local Expertscited by 3
  16. 2021Max-Linear Regression by Convex Programmingcited by 2
  17. 2021Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networkscited by 2
  18. 2021Sparsest Univariate Learning Models Under Lipschitz Constraintcited by 2
  19. 2022Manifoldron: Direct Space Partition via Manifold Discoverycited by 2
  20. 2024TTVD: Towards a Geometric Framework for Test-Time Adaptation Based on Voronoi Diagramcited by 2
  21. 2022Over-parameterization: A Necessary Condition for Models that Extrapolatecited by 1
  22. 2023Neural Polytopescited by 1
  23. 2024Mitigating over-exploration in latent space optimization using LEScited by 1
  24. 2025Computing Linear Regions in Neural Networks with Skip Connectionscited by 1
  25. 2025Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusioncited by 1
  26. 2025Concept Attractors in LLMs and their Applicationscited by 1
  27. 2026Characterizing the Discrete Geometry of ReLU Networkscited by 1
  28. 2021Learning DNN networks using un-rectifying ReLU with compressed sensing applicationcited by 0
  29. 2021Polynomial-Spline Neural Networks with Exact Integralscited by 0
  30. 2022Towards Global Neural Network Abstractions with Locally-Exact Reconstructioncited by 0
  31. 2023SpecXAI -- Spectral interpretability of Deep Learning Modelscited by 0
  32. 2023Unsupervised Discovery of Extreme Weather Events Using Universal Representations of Emergent Organizationcited by 0
  33. 2023A max-affine spline approximation of neural networks using the Legendre transform of a convex-concave representationcited by 0
  34. 2024Towards stable training of parallel continual learningcited by 0
  35. 2024On the Geometry of Deep Learningcited by 0
  36. 2025Semialgebraic Neural Networks: From roots to representationscited by 0
  37. 2025A Priori Generalizability Estimate for a CNNcited by 0
  38. 2025Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activationscited by 0

50 most cited of 55 All 55 (JSON)