# Accelerating Minibatch Stochastic Gradient Descent using Typicality Sampling

arXiv: https://arxiv.org/abs/1903.04192 · HTML: https://mlsanity.com/paper/1903.04192/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2019.

## Built on by (most cited first, 11 of 11)

- [Learning in PINNs: Phase transition, diffusion equilibrium, and generalization](https://mlsanity.com/paper/2403.18494/) (2024)
- [A Survey of Dataset Refinement for Problems in Computer Vision Datasets](https://mlsanity.com/paper/2210.11717/) (2022)
- [First-Passage Approach to Optimizing Perturbations for Improved Training of Machine Learning Models](https://mlsanity.com/paper/2502.04121/) (2025)
- [Variance Matters: Improving Domain Adaptation via Stratified Sampling](https://mlsanity.com/paper/2512.05226/) (2025)
- [Order Matters: Improving Domain Adaptation by Reordering Data](https://mlsanity.com/paper/2605.05084/) (2026)
- [Understanding Short-Range Memory Effects in Deep Neural Networks](https://mlsanity.com/paper/2105.02062/) (2021)
- [Drill the Cork of Information Bottleneck by Inputting the Most Important Data](https://mlsanity.com/paper/2105.07181/) (2021)
- [Transfer Learning with Active Sampling for Rapid Training and Calibration in BCI-P300 Across Health States and Multi-centre Data](https://mlsanity.com/paper/2412.17833/) (2024)
- [Some like it tough: Improving model generalization via progressively increasing the training difficulty](https://mlsanity.com/paper/2110.13058/) (2021)
- [Water Quality Data Imputation via A Fast Latent Factorization of Tensors with PID-based Optimizer](https://mlsanity.com/paper/2503.06997/) (2025)
- [Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGD](https://mlsanity.com/paper/2505.18535/) (2025)

## Abstract (arXiv)

Machine learning, especially deep neural networks, has been rapidly developed in fields including computer vision, speech recognition and reinforcement learning. Although Mini-batch SGD is one of the most popular stochastic optimization methods in training deep networks, it shows a slow convergence rate due to the large noise in gradient approximation. In this paper, we attempt to remedy this problem by building more efficient batch selection method based on typicality sampling, which reduces the error of gradient estimation in conventional Minibatch SGD. We analyze the convergence rate of the resulting typical batch SGD algorithm and compare convergence properties between Minibatch SGD and the algorithm. Experimental results demonstrate that our batch selection scheme works well and more complex Minibatch SGD variants can benefit from the proposed batch selection strategy.