mlsanityData to 7 Oct 2026

Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks

arXiv 2102.06604 ↗First posted 12 Feb 2021cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

When engineers train deep learning models, they are very much 'flying blind'. Commonly used methods for real-time training diagnostics, such as monitoring the train/test loss, are limited. Assessing a network's training process solely through these performance indicators is akin to debugging software without access to internal states through a debugger. To address this, we present Cockpit, a collection of instruments that enable a closer look into the inner workings of a learning machine, and a more informative and meaningful status report for practitioners. It facilitates the identification of learning phases and failure modes, like ill-chosen hyperparameters. These instruments leverage novel higher-order information about the gradient distribution and curvature, which has only recently become efficiently accessible. We believe that such a debugging tool, which we open-source for PyTorch, is a valuable help in troubleshooting the training process. By revealing new insights, it also more generally contributes to explainability and interpretability of deep nets.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

34 of 52 references.

References · 34

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  7. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  9. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  10. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  11. 2020The Computational Limits of Deep Learningcited by 188
  12. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
22 more
  1. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  2. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  3. 2020The Early Phase of Neural Network Trainingcited by 114
  4. 2019BackPACK: Packing more into backpropcited by 77
  5. 2018A Walk with SGDcited by 74
  6. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  7. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
  8. 2019Generalization in Deep Networks: The Role of Distance from Initializationcited by 49
  9. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  10. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  11. 2020Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimizationcited by 41
  12. 2020Understanding Why Neural Networks Generalize Well Through GSNR of Parameterscited by 40
  13. 2017Early Stopping without a Validation Setcited by 39
  14. 2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
  15. 2015Probabilistic Line Searches for Stochastic Optimizationcited by 29
  16. 2016Coupling Adaptive Batch Sizes with Learning Ratescited by 29
  17. 2017Adaptive Sampling Strategies for Stochastic Optimizationcited by 29
  18. 2020A Study of Gradient Variance in Deep Learningcited by 27
  19. 2020Making Coherence Out of Nothing At All: Measuring the Evolution of Gradient Alignmentcited by 7
  20. 2019A Dynamic Sampling Adaptive-SGD Method for Machine Learningcited by 2
  21. 2020Investigating Learning in Deep Neural Networks using Layer-Wise Weight Changecited by 2
  22. 2020On regularization of gradient descent, layer imbalance and flat minimacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 0; 2023: 1; 2024: 0; 2025: 2; 2026: 3.

7 papers.

  1. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  2. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  3. 2023Going Further: Flatness at the Rescue of Early Stopping for Adversarial Example Transferabilitycited by 2
  4. 2025Per-example gradients: a new frontier for understanding and improving optimizerscited by 2
  5. 2026BTTackler: A Diagnosis-based Framework for Efficient Deep Learning Hyperparameter Optimizationcited by 0
  6. 2026Second-Order, First-Class: A Composable Stack for Curvature-Aware Trainingcited by 0
  7. 2026Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programscited by 0