# Approximate Fisher Information Matrix to Characterise the Training of Deep Neural Networks

arXiv: https://arxiv.org/abs/1810.06767 · HTML: https://mlsanity.com/paper/1810.06767/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2018.

## Built on by (most cited first, 12 of 12)

- [Weighted Ensemble Models Are Strong Continual Learners](https://mlsanity.com/paper/2312.08977/) (2023)
- [Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics](https://mlsanity.com/paper/2106.00734/) (2021)
- [NoT: Federated Unlearning via Weight Negation](https://mlsanity.com/paper/2503.05657/) (2025)
- [Effective dimension of machine learning models](https://mlsanity.com/paper/2112.04807/) (2021)
- [Evolutionary Optimization for Designing Variational Quantum Circuits with High Model Capacity](https://mlsanity.com/paper/2412.12484/) (2024)
- [MAGICS: Adversarial RL with Minimax Actors Guided by Implicit Critic Stackelberg for Convergent Neural Synthesis of Robot Safety](https://mlsanity.com/paper/2409.13867/) (2024)
- [Efficient Activation Function Optimization through Surrogate Modeling](https://mlsanity.com/paper/2301.05785/) (2023)
- [Task2Box: Box Embeddings for Modeling Asymmetric Task Relationships](https://mlsanity.com/paper/2403.17173/) (2024)
- [Information Guided Regularization for Fine-tuning Language Models](https://mlsanity.com/paper/2406.14005/) (2024)
- [T2I-ConBench: Text-to-Image Benchmark for Continual Post-training](https://mlsanity.com/paper/2505.16875/) (2025)
- [Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes](https://mlsanity.com/paper/2511.17399/) (2025)
- [TASER: Task-Aware Spectral Energy Refine for Backdoor Suppression in UAV Swarms Decentralized Federated Learning](https://mlsanity.com/paper/2603.10075/) (2026)

## Abstract (arXiv)

In this paper, we introduce a novel methodology for characterising the performance of deep learning networks (ResNets and DenseNet) with respect to training convergence and generalisation as a function of mini-batch size and learning rate for image classification. This methodology is based on novel measurements derived from the eigenvalues of the approximate Fisher information matrix, which can be efficiently computed even for high capacity deep models. Our proposed measurements can help practitioners to monitor and control the training process (by actively tuning the mini-batch size and learning rate) to allow for good training convergence and generalisation. Furthermore, the proposed measurements also allow us to show that it is possible to optimise the training process with a new dynamic sampling training approach that continuously and automatically change the mini-batch size and learning rate during the training process. Finally, we show that the proposed dynamic sampling training approach has a faster training time and a competitive classification accuracy compared to the current state of the art.