# Online Deterministic Annealing for Classification and Clustering

arXiv: https://arxiv.org/abs/2102.05836 · HTML: https://mlsanity.com/paper/2102.05836/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 69 references that are in the map.
- First posted on arXiv 11 Feb 2021.

## Builds on (most cited first, 12 of 12)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020)
- [Adversarial Training Can Hurt Generalization](https://mlsanity.com/paper/1906.06032/) (2019)
- [Deep Continuous Clustering](https://mlsanity.com/paper/1803.01449/) (2018)
- [Centroid-based deep metric learning for speaker recognition](https://mlsanity.com/paper/1902.02375/) (2019)
- [Persistence Codebooks for Topological Data Analysis](https://mlsanity.com/paper/1802.04852/) (2018)
- [Using KL-divergence to focus Deep Visual Explanation](https://mlsanity.com/paper/1711.06431/) (2017)
- [Robustness of Generalized Learning Vector Quantization Models against Adversarial Attacks](https://mlsanity.com/paper/1902.00577/) (2019)
- [Prototype-based Neural Network Layers: Incorporating Vector Quantization](https://mlsanity.com/paper/1812.01214/) (2018)

## Built on by (most cited first, 6 of 6)

- [Annealing Optimization for Progressive Learning with Stochastic Approximation](https://mlsanity.com/paper/2209.02826/) (2022)
- [Multi-Resolution Online Deterministic Annealing: A Hierarchical and Progressive Learning Architecture](https://mlsanity.com/paper/2212.08189/) (2022)
- [Spiking Neural Network Feature Discrimination Boosts Modality Fusion](https://mlsanity.com/paper/2502.10423/) (2025)
- [Towards the One Learning Algorithm Hypothesis: A System-theoretic Approach](https://mlsanity.com/paper/2112.02256/) (2021)
- [Real-time Hybrid System Identification with Online Deterministic Annealing](https://mlsanity.com/paper/2408.01730/) (2024)
- [Autonomy-Aware Clustering: When Local Decisions Supersede Global Prescriptions](https://mlsanity.com/paper/2509.25775/) (2025)

## Abstract (arXiv)

Inherent in virtually every iterative machine learning algorithm is the problem of hyper-parameter tuning, which includes three major design parameters: (a) the complexity of the model, e.g., the number of neurons in a neural network, (b) the initial conditions, which heavily affect the behavior of the algorithm, and (c) the dissimilarity measure used to quantify its performance. We introduce an online prototype-based learning algorithm that can be viewed as a progressively growing competitive-learning neural network architecture for classification and clustering. The learning rule of the proposed approach is formulated as an online gradient-free stochastic approximation algorithm that solves a sequence of appropriately defined optimization problems, simulating an annealing process. The annealing nature of the algorithm contributes to avoiding poor local minima, offers robustness with respect to the initial conditions, and provides a means to progressively increase the complexity of the learning model, through an intuitive bifurcation phenomenon. The proposed approach is interpretable, requires minimal hyper-parameter tuning, and allows online control over the performance-complexity trade-off. Finally, we show that Bregman divergences appear naturally as a family of dissimilarity measures that play a central role in both the performance and the computational complexity of the learning algorithm.