# Scalar Representations of Neural Network Training Dynamics

arXiv: https://arxiv.org/abs/2606.30384 · HTML: https://mlsanity.com/paper/2606.30384/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 23 of 23)

- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes](https://mlsanity.com/paper/1706.10239/) (2017)
- [Classifying the classifier: dissecting the weight space of neural networks](https://mlsanity.com/paper/2002.05688/) (2020)
- [Does SGD really happen in tiny subspaces?](https://mlsanity.com/paper/2405.16002/) (2024)
- [What can linear interpolation of neural network loss landscapes tell us?](https://mlsanity.com/paper/2106.16004/) (2021)
- [Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"](https://mlsanity.com/paper/2012.06898/) (2020)
- [The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold](https://mlsanity.com/paper/2305.01604/) (2023)
- [A Survey of Weight Space Learning: Understanding, Representation, and Generation](https://mlsanity.com/paper/2603.10090/) (2026)
- [Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition](https://mlsanity.com/paper/2408.08944/) (2024)
- [Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions](https://mlsanity.com/paper/1901.02302/) (2019)
- [Finite-time Lyapunov exponents of deep neural networks](https://mlsanity.com/paper/2306.12548/) (2023)
- [PCA of high dimensional random walks with comparison to neural network training](https://mlsanity.com/paper/1806.08805/) (2018)
- [Dynamical stability and chaos in artificial neural network trajectories along training](https://mlsanity.com/paper/2404.05782/) (2024)
- [Correlations of network trajectories](https://mlsanity.com/paper/2206.03334/) (2022)
- [Scalar embedding of temporal network trajectories](https://mlsanity.com/paper/2412.02715/) (2024)
- [Leveraging chaotic transients in the training of artificial neural networks](https://mlsanity.com/paper/2506.08523/) (2025)
- [Predictability of temporal network dynamics in normal ageing and brain pathology](https://mlsanity.com/paper/2502.16557/) (2025)

## Abstract (arXiv)

Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape. However, the large number of trainable parameters makes the direct analysis of these dynamics challenging. In this work, we treat such training trajectories as temporal networks and apply recently proposed strategies for the scalar embedding of temporal networks. We investigate whether such a scalar embedding provides a meaningful low-dimensional representation of neural network training dynamics. Using a multilayer perceptron trained on the MNIST classification task, we show that the embedding preserves the main dynamical features observed in the original parameter space, including the emergence of sensitivity to initial conditions for specific learning rate regimes and an accurate reconstruction of the network's maximum Lyapunov exponent. We then use the embedded scalar trajectory to define a characteristic time, analogous to a Lyapunov time, after which the exponential separation between initially close embedded trajectories saturates. This characteristic time captures the typical decorrelation time between initially close network trajectories in the original high-dimensional system. Finally, we investigate the statistical organization of asymptotic training states through a spacing observable defined in the embedded space. We find that the distributions of rescaled asymptotic spacings collapse onto a common form across initial conditions and are compatible with a skew lognormal distribution. Altogether, our results suggest that scalar low-dimensional embeddings provide a useful framework for studying and visualizing the dynamical properties of neural network optimization trajectories.