# Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)

arXiv: https://arxiv.org/abs/1604.02313 · HTML: https://mlsanity.com/paper/1604.02313/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2016.

## Built on by (most cited first, 16 of 16)

- [Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective](https://mlsanity.com/paper/2210.01787/) (2022)
- [A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness](https://mlsanity.com/paper/2205.00403/) (2022)
- [Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networks](https://mlsanity.com/paper/2208.03160/) (2022)
- [Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport?](https://mlsanity.com/paper/2206.07767/) (2022)
- [Approximation of Lipschitz Functions using Deep Spline Neural Networks](https://mlsanity.com/paper/2204.06233/) (2022)
- [Normalization-Equivariant Neural Networks with Application to Image Denoising](https://mlsanity.com/paper/2306.05037/) (2023)
- [Approximation theory for 1-Lipschitz ResNets](https://mlsanity.com/paper/2505.12003/) (2025)
- [Convolutional Neural Networks as 2-D systems](https://mlsanity.com/paper/2303.03042/) (2023)
- [Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Loss](https://mlsanity.com/paper/2505.15174/) (2025)
- [Robust-by-Design Classification via Unitary-Gradient Neural Networks](https://mlsanity.com/paper/2209.04293/) (2022)
- [A Wasserstein perspective of Vanilla GANs](https://mlsanity.com/paper/2403.15312/) (2024)
- [Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operators](https://mlsanity.com/paper/2110.11940/) (2021)
- [Computing the Distance between unbalanced Distributions -- The flat Metric](https://mlsanity.com/paper/2308.01039/) (2023)
- [Preserving Plasticity in Continual Learning via Dynamical Isometry](https://mlsanity.com/paper/2606.09762/) (2026)
- [Adaptive n-ary Activation Functions for Probabilistic Boolean Logic](https://mlsanity.com/paper/2203.08977/) (2022)
- [Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches, and Certifiable Robustness](https://mlsanity.com/paper/2607.16329/) (2026)

## Abstract (arXiv)

We propose a novel activation function that implements piece-wise orthogonal non-linear mappings based on permutations. It is straightforward to implement, and very computationally efficient, also it has little memory requirements. We tested it on two toy problems for feedforward and recurrent networks, it shows similar performance to tanh and ReLU. OPLU activation function ensures norm preservance of the backpropagated gradients, therefore it is potentially good for the training of deep, extra deep, and recurrent neural networks.