# LCA: Loss Change Allocation for Neural Network Training

arXiv: https://arxiv.org/abs/1909.01440 · HTML: https://mlsanity.com/paper/1909.01440/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2019.

## Built on by (most cited first, 9 of 9)

- [Rethinking Large Language Models in Mental Health Applications](https://mlsanity.com/paper/2311.11267/) (2023)
- [Hidden Breakthroughs in Language Model Training](https://mlsanity.com/paper/2506.15872/) (2025)
- [GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization](https://mlsanity.com/paper/2210.06895/) (2022)
- [Exploring Data Redundancy in Real-world Image Classification through Data Selection](https://mlsanity.com/paper/2306.14113/) (2023)
- [ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behavior](https://mlsanity.com/paper/2505.20076/) (2025)
- [Experiments with Rich Regime Training for Deep Learning](https://mlsanity.com/paper/2102.13522/) (2021)
- [Adversarial Parameter Defense by Multi-Step Risk Minimization](https://mlsanity.com/paper/2109.02889/) (2021)
- [Gradient Mask: Lateral Inhibition Mechanism Improves Performance in Artificial Neural Networks](https://mlsanity.com/paper/2208.06918/) (2022)
- [Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models](https://mlsanity.com/paper/2609.34933/) (2026)

## Abstract (arXiv)

Neural networks enjoy widespread use, but many aspects of their training, representation, and operation are poorly understood. In particular, our view into the training process is limited, with a single scalar loss being the most common viewport into this high-dimensional, dynamic process. We propose a new window into training called Loss Change Allocation (LCA), in which credit for changes to the network loss is conservatively partitioned to the parameters. This measurement is accomplished by decomposing the components of an approximate path integral along the training trajectory using a Runge-Kutta integrator. This rich view shows which parameters are responsible for decreasing or increasing the loss during training, or which parameters "help" or "hurt" the network's learning, respectively. LCA may be summed over training iterations and/or over neurons, channels, or layers for increasingly coarse views. This new measurement device produces several insights into training. (1) We find that barely over 50% of parameters help during any given iteration. (2) Some entire layers hurt overall, moving on average against the training gradient, a phenomenon we hypothesize may be due to phase lag in an oscillatory training process. (3) Finally, increments in learning proceed in a synchronized manner across layers, often peaking on identical iterations.