# Identifying Equivalent Training Dynamics arXiv: https://arxiv.org/abs/2302.09160 ยท HTML: https://mlsanity.com/paper/2302.09160/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 35 of its 76 references that are in the map. - First posted on arXiv 17 Feb 2023. ## Builds on (most cited first, 30 of 35) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016) - [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019) - [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018) - [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019) - [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022) - [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020) - [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022) - [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020) - [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021) - [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018) - [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022) - [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020) - [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019) - [Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints](https://mlsanity.com/paper/1408.3595/) (2014) - [Critical Learning Periods in Deep Neural Networks](https://mlsanity.com/paper/1711.08856/) (2017) - [Extracting spatial-temporal coherent patterns in large-scale neural recordings using dynamic mode decomposition](https://mlsanity.com/paper/1409.5496/) (2014) - [Rigorous data-driven computation of spectral properties of Koopman operators for dynamical systems](https://mlsanity.com/paper/2111.14889/) (2021) - [Optimizing Neural Networks via Koopman Operator Theory](https://mlsanity.com/paper/2006.02361/) (2020) - [The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion](https://mlsanity.com/paper/2107.09133/) (2021) - [An Operator Theoretic View on Pruning Deep Neural Networks](https://mlsanity.com/paper/2110.14856/) (2021) - [Applications of Koopman Mode Analysis to Neural Networks](https://mlsanity.com/paper/2006.11765/) (2020) - [Non-convex online learning via algorithmic equivalence](https://mlsanity.com/paper/2205.15235/) (2022) - [Personalized Algorithm Generation: A Case Study in Learning ODE Integrators](https://mlsanity.com/paper/2105.01303/) (2021) - [On the generalization of learning algorithms that do not converge](https://mlsanity.com/paper/2208.07951/) (2022) - [Algorithmic (Semi-)Conjugacy via Koopman Operator Theory](https://mlsanity.com/paper/2209.06374/) (2022) - [A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms](https://mlsanity.com/paper/2211.12386/) (2022) - [Accelerating Training in Artificial Neural Networks with Dynamic Mode Decomposition](https://mlsanity.com/paper/2006.14371/) (2020) ## Built on by (most cited first, 9 of 9) - [The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions](https://mlsanity.com/paper/2506.13234/) (2025) - [InputDSA: Demixing then Comparing Recurrent and Externally Driven Dynamics](https://mlsanity.com/paper/2510.25943/) (2025) - [A Spectral-Grassmann Wasserstein metric for operator representations of dynamical systems](https://mlsanity.com/paper/2509.24920/) (2025) - [Distributional Spectral Diagnostics for Localizing Grokking Transitions](https://mlsanity.com/paper/2605.08237/) (2026) - [Finding separatrices of dynamical flows with Deep Koopman Eigenfunctions](https://mlsanity.com/paper/2505.15231/) (2025) - [Koopman operator theory: fundamentals, control, and applications](https://mlsanity.com/paper/2607.01819/) (2026) - [Fast dynamical similarity analysis](https://mlsanity.com/paper/2511.22828/) (2025) - [Interpreting Reinforcement Learning Model Behavior via Koopman with Control](https://mlsanity.com/paper/2603.19968/) (2026) - [Temporal Geometry of Deep Networks: Hyperbolic Representations of Training Dynamics for Intrinsic Explainability](https://mlsanity.com/paper/2610.03000/) (2026) ## Abstract (arXiv) Study of the nonlinear evolution deep neural network (DNN) parameters undergo during training has uncovered regimes of distinct dynamical behavior. While a detailed understanding of these phenomena has the potential to advance improvements in training efficiency and robustness, the lack of methods for identifying when DNN models have equivalent dynamics limits the insight that can be gained from prior work. Topological conjugacy, a notion from dynamical systems theory, provides a precise definition of dynamical equivalence, offering a possible route to address this need. However, topological conjugacies have historically been challenging to compute. By leveraging advances in Koopman operator theory, we develop a framework for identifying conjugate and non-conjugate training dynamics. To validate our approach, we demonstrate that comparing Koopman eigenvalues can correctly identify a known equivalence between online mirror descent and online gradient descent. We then utilize our approach to: (a) identify non-conjugate training dynamics between shallow and wide fully connected neural networks; (b) characterize the early phase of training dynamics in convolutional neural networks; (c) uncover non-conjugate training dynamics in Transformers that do and do not undergo grokking. Our results, across a range of DNN architectures, illustrate the flexibility of our framework and highlight its potential for shedding new light on training dynamics.