mlsanityData to 7 Oct 2026

Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks

arXiv 2402.01344 ↗First posted 2 Feb 2024cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

This paper presents a new bi-Lipschitz invertible neural network, the BiLipNet, which has the ability to smoothly control both its Lipschitzness (output sensitivity to input perturbations) and inverse Lipschitzness (input distinguishability from different outputs). The second main contribution is a new scalar-output network, the PLNet, which is a composition of a BiLipNet and a quadratic potential. We show that PLNet satisfies the Polyak-Lojasiewicz condition and can be applied to learn non-convex surrogate losses with a unique and efficiently-computable global minimum. The central technical element in these networks is a novel invertible residual layer with certified strong monotonicity and Lipschitzness, which we compose with orthogonal layers to build the BiLipNet. The certification of these properties is based on incremental quadratic constraints, resulting in much tighter bounds than can be achieved with spectral normalization. Moreover, we formulate the calculation of the inverse of a BiLipNet -- and hence the minimum of a PLNet -- as a series of three-operator splitting problems, for which fast algorithms can be applied.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 47 of its 70 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

47 of 70 references.

References · 47

  1. 2017Improved Training of Wasserstein GANscited by 2,223
  2. 2016Density estimation using Real NVPcited by 2,029
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  5. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  6. 2019Deep Equilibrium Modelscited by 603
  7. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  8. 2016Input Convex Neural Networkscited by 407
  9. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  10. 2018Invertible Residual Networkscited by 364
  11. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  12. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
35 more
  1. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 258
  2. 2019Residual Flows for Invertible Generative Modelingcited by 256
  3. 2018Analyzing Inverse Problems with Invertible Neural Networkscited by 239
  4. 2018Neural Autoregressive Flowscited by 207
  5. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  6. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  7. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  8. 2019Implicit Deep Learningcited by 124
  9. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  10. 2020Monotone operator equilibrium networkscited by 100
  11. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  12. 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 79
  13. 2020Understanding and Mitigating Exploding Inverses in Invertible Neural Networkscited by 71
  14. 2021Skew Orthogonal Convolutionscited by 67
  15. 2019Block Neural Autoregressive Flowcited by 62
  16. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  17. 2023A Unified Algebraic Perspective on Lipschitz Neural Networkscited by 58
  18. 2018Resampled Priors for Variational Autoencoderscited by 56
  19. 2020Lipschitz Bounded Equilibrium Networkscited by 49
  20. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  21. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 49
  22. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  23. 2021Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustnesscited by 43
  24. 2021Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neuronscited by 40
  25. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  26. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 37
  27. 2015A Three-Operator Splitting Scheme and its Optimization Applicationscited by 34
  28. 2021Implicit Normalizing Flowscited by 28
  29. 2019CAQL: Continuous Action Q-Learningcited by 22
  30. 2021Invertible DenseNets with Concatenated LipSwishcited by 17
  31. 2022A Quantitative Geometric Approach to Neural-Network Smoothnesscited by 16
  32. 2018Lifted Proximal Operator Machinescited by 14
  33. 2024Functional Graphical Models: Structure Enables Offline Data-Driven Optimizationcited by 11
  34. 2022Invertible Monotone Operators for Normalizing Flowscited by 6
  35. 2024Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 4.

9 papers.

  1. 2024Learning Stable and Passive Neural Differential Equationscited by 2
  2. 2024LipKernel: Lipschitz-Bounded Convolutional Neural Networks via Dissipative Layerscited by 2
  3. 2025R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networkscited by 2
  4. 2025Robustly Invertible Nonlinear Dynamics and the BiLipREN: Contracting Neural Models with Contracting Inversescited by 1
  5. 2025From Sublinear to Linear: Local Convergence in Finite-Width Networks via Locally Polyak-Lojasiewicz Regionscited by 1
  6. 2026Controller Design for Structured State-space Models via Contraction Theorycited by 1
  7. 2026Monotonic Kolmogorov-Arnold Networks: A Theoretical and Empirical Study of Monotonicity as an Inductive Biascited by 1
  8. 2026Online Continual Learning for Time Series: a Natural Score-driven Approachcited by 0
  9. 2026Robustly Invertible Nonlinear Dynamics and the BiLipREN: From Inversion-Based Control to Generative Trajectory Modellingcited by 0