# D3O: Dynamic Distribution Distillation for Ordinal Regression

arXiv: https://arxiv.org/abs/2607.23575 · HTML: https://mlsanity.com/paper/2607.23575/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Photo Aesthetics Ranking Network with Attributes and Content Adaptation](https://mlsanity.com/paper/1606.01621/) (2016)
- [Label Distribution Learning](https://mlsanity.com/paper/1408.6027/) (2014)
- [Deep Label Distribution Learning with Label Ambiguity](https://mlsanity.com/paper/1611.01731/) (2016)
- [Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression](https://mlsanity.com/paper/2103.13629/) (2021)
- [Personalized Image Aesthetics Assessment with Rich Attributes](https://mlsanity.com/paper/2203.16754/) (2022)
- [OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression](https://mlsanity.com/paper/2206.02338/) (2022)
- [BridgeNet: A Continuity-Aware Probabilistic Network for Age Estimation](https://mlsanity.com/paper/1904.03358/) (2019)
- [PML: Progressive Margin Loss for Long-tailed Age Classification](https://mlsanity.com/paper/2103.02140/) (2021)
- [MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition](https://mlsanity.com/paper/2208.05768/) (2022)
- [Unimodal probability distributions for deep ordinal classification](https://mlsanity.com/paper/1705.05278/) (2017)
- [An Image is Worth More than a Thousand Favorites: Surfacing the Hidden Beauty of Flickr Pictures](https://mlsanity.com/paper/1505.03358/) (2015)
- [Ord2Seq: Regarding Ordinal Regression as Label Sequence Prediction](https://mlsanity.com/paper/2307.09004/) (2023)
- [CLOC: Contrastive Learning for Ordinal Classification with Multi-Margin N-pair Loss](https://mlsanity.com/paper/2504.17813/) (2025)
- [Spatial Self-Distillation for Object Detection with Inaccurate Bounding Boxes](https://mlsanity.com/paper/2307.12101/) (2023)
- [A Survey on Ordinal Regression: Applications, Advances and Prospects](https://mlsanity.com/paper/2503.00952/) (2025)
- [Learning multiple non-mutually-exclusive tasks for improved classification of inherently ordered labels](https://mlsanity.com/paper/1805.11837/) (2018)
- [Label Information Bottleneck for Label Enhancement](https://mlsanity.com/paper/2303.06836/) (2023)
- [Ordinal Label-Distribution Learning with Constrained Asymmetric Priors for Imbalanced Retinal Grading](https://mlsanity.com/paper/2509.26146/) (2025)
- [Extract More from Less: Efficient Fine-Grained Visual Recognition in Low-Data Regimes](https://mlsanity.com/paper/2406.19814/) (2024)
- [Dual-level Fuzzy Learning with Patch Guidance for Image Ordinal Regression](https://mlsanity.com/paper/2505.05834/) (2025)

## Abstract (arXiv)

Ordinal regression is widely used in scenarios where labels are discrete yet inherently ordered. In practice, however, ordinal labels are often obtained by discretizing underlying continuous semantics through subjective human judgment, resulting in ambiguous boundaries and annotation noise. Such uncertainty challenges existing methods that rely on fixed supervision targets, which may reinforce biased ordering under subjective annotations. To address this limitation, we propose D3O, a dynamic distribution distillation framework that replaces static supervision with training-driven evolution of ordinal label distributions via self-distillation. Specifically, we introduce a contrastive ordinal-aware label enhancement module that leverages vision-language alignment to recover refined label distributions capturing both inter-class ambiguity and instance-level uncertainty. Furthermore, we design a CDF-based cross-layer interaction distillation mechanism to propagate cumulative ordinal structure across network hierarchy, ensuring consistent ordinal geometry in intermediate representations. Extensive experiments on four general ordinal regression tasks demonstrate that our proposed D3O consistently outperforms existing approaches, particularly under severe class imbalance and noisy supervision. These results highlight the effectiveness of dynamic supervision in learning robust ordinal representations beyond fixed targets. The code will be publicly available.