# On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions

arXiv: https://arxiv.org/abs/2503.21708 · HTML: https://mlsanity.com/paper/2503.21708/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 27 Mar 2025.

## Builds on (most cited first, 10 of 10)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Characterizing signal propagation to close the performance gap in unnormalized ResNets](https://mlsanity.com/paper/2101.08692/) (2021)
- [Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction](https://mlsanity.com/paper/2206.07085/) (2022)
- [On the Nonlinearity of Layer Normalization](https://mlsanity.com/paper/2406.01255/) (2024)
- [Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)](https://mlsanity.com/paper/1710.09967/) (2017)

## Built on by (most cited first, 4 of 4)

- [Stronger Normalization-Free Transformers](https://mlsanity.com/paper/2512.10938/) (2025)
- [SeeDNorm: Self-Rescaled Dynamic Normalization](https://mlsanity.com/paper/2510.22777/) (2025)
- [Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms](https://mlsanity.com/paper/2605.08423/) (2026)
- [Sphere Retraction Normalizations](https://mlsanity.com/paper/2608.02668/) (2026)

## Abstract (arXiv)

Layer normalization (LN) is an essential component of modern neural networks. While many alternative techniques have been proposed, none of them have succeeded in replacing LN so far. The latest suggestion in this line of research is a dynamic activation function called Dynamic Tanh (DyT). Although it is empirically well-motivated and appealing from a practical point of view, it lacks a theoretical foundation. In this work, we shed light on the mathematical relationship between LN and dynamic activation functions. In particular, we derive DyT from the LN variant RMSNorm, and show that a well-defined decoupling in derivative space as well as an approximation are needed to do so. By applying the same decoupling procedure directly in function space, we are able to omit the approximation and obtain the exact element-wise counterpart of RMSNorm, which we call Dynamic Inverse Square Root Unit (DyISRU). We demonstrate numerically that DyISRU reproduces the normalization effect on outliers more accurately than DyT does.