# Multi-Resolution Online Deterministic Annealing: A Hierarchical and Progressive Learning Architecture

arXiv: https://arxiv.org/abs/2212.08189 · HTML: https://mlsanity.com/paper/2212.08189/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 40 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 8 of 8)

- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020)
- [Understanding Deep Convolutional Networks](https://mlsanity.com/paper/1601.04920/) (2016)
- [Kymatio: Scattering Transforms in Python](https://mlsanity.com/paper/1812.11214/) (2018)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Deep Convolutional Networks are Hierarchical Kernel Machines](https://mlsanity.com/paper/1508.01084/) (2015)
- [Online Deterministic Annealing for Classification and Clustering](https://mlsanity.com/paper/2102.05836/) (2021)
- [Annealing Optimization for Progressive Learning with Stochastic Approximation](https://mlsanity.com/paper/2209.02826/) (2022)

## Built on by (most cited first, 1 of 1)

- [Real-time Hybrid System Identification with Online Deterministic Annealing](https://mlsanity.com/paper/2408.01730/) (2024)

## Abstract (arXiv)

Hierarchical learning algorithms that gradually approximate a solution to a data-driven optimization problem are essential to decision-making systems, especially under limitations on time and computational resources. In this study, we introduce a general-purpose hierarchical learning architecture that is based on the progressive partitioning of a possibly multi-resolution data space. The optimal partition is gradually approximated by solving a sequence of optimization sub-problems that yield a sequence of partitions with increasing number of subsets. We show that the solution of each optimization problem can be estimated online using gradient-free stochastic approximation updates. As a consequence, a function approximation problem can be defined within each subset of the partition and solved using the theory of two-timescale stochastic approximation algorithms. This simulates an annealing process and defines a robust and interpretable heuristic method to gradually increase the complexity of the learning architecture in a task-agnostic manner, giving emphasis to regions of the data space that are considered more important according to a predefined criterion. Finally, by imposing a tree structure in the progression of the partitions, we provide a means to incorporate potential multi-resolution structure of the data space into this approach, significantly reducing its complexity, while introducing hierarchical variable-rate feature extraction properties similar to certain classes of deep learning architectures. Asymptotic convergence analysis and experimental results are provided for supervised and unsupervised learning problems.