# Learning compositional functions via multiplicative weight updates

arXiv: https://arxiv.org/abs/2006.14560 · HTML: https://mlsanity.com/paper/2006.14560/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2020.

## Built on by (most cited first, 20 of 20)

- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [A Spectral Condition for Feature Learning](https://mlsanity.com/paper/2310.17813/) (2023)
- [Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization](https://mlsanity.com/paper/2503.12645/) (2025)
- [Weight Clipping for Deep Continual and Reinforcement Learning](https://mlsanity.com/paper/2407.01704/) (2024)
- [Learning by Turning: Neural Architecture Aware Optimisation](https://mlsanity.com/paper/2102.07227/) (2021)
- [An Embedding of ReLU Networks and an Analysis of their Identifiability](https://mlsanity.com/paper/2107.09370/) (2021)
- [Adaptive Gradient Methods with Local Guarantees](https://mlsanity.com/paper/2203.01400/) (2022)
- [LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Update](https://mlsanity.com/paper/2106.13914/) (2021)
- [Multiplicative Reweighting for Robust Neural Network Optimization](https://mlsanity.com/paper/2102.12192/) (2021)
- [The Lie-Group Bayesian Learning Rule](https://mlsanity.com/paper/2303.04397/) (2023)
- [Graph Matching via convex relaxation to the simplex](https://mlsanity.com/paper/2310.20609/) (2023)
- [Accelerating Hopfield Network Dynamics: Beyond Synchronous Updates and Forward Euler](https://mlsanity.com/paper/2311.15673/) (2023)
- [Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks](https://mlsanity.com/paper/2506.17768/) (2025)
- [Multiplicative update rules for accelerating deep learning training and increasing robustness](https://mlsanity.com/paper/2307.07189/) (2023)
- [M+Adam: Low-Precision Training via Additive-Multiplicative Optimization](https://mlsanity.com/paper/2607.10611/) (2026)
- [Probabilistic Trust Intervals for Out of Distribution Detection](https://mlsanity.com/paper/2102.01336/) (2021)
- [Boosting Certified ℓ_∞ Robustness with EMA Method and Ensemble Model](https://mlsanity.com/paper/2107.00230/) (2021)
- [Online-Learning Deep Neuro-Adaptive Dynamic Inversion Controller for Model Free Control](https://mlsanity.com/paper/2107.10383/) (2021)
- [Non-negative isomorphic neural networks for photonic neuromorphic accelerators](https://mlsanity.com/paper/2310.01084/) (2023)
- [HORST: Composing Optimizer Geometries for Sparse Transformer Training](https://mlsanity.com/paper/2605.21104/) (2026)

## Abstract (arXiv)

Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam -- a multiplicative version of the Adam optimiser -- and show that it can train state of the art neural network architectures without learning rate tuning. We further show that Madam is easily adapted to train natively compressed neural networks by representing their weights in a logarithmic number system. We conclude by drawing connections between multiplicative weight updates and recent findings about synapses in biology.