# R-divergence for Estimating Model-oriented Distribution Discrepancy

arXiv: https://arxiv.org/abs/2310.01109 · HTML: https://mlsanity.com/paper/2310.01109/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 63 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Are Anchor Points Really Indispensable in Label-Noise Learning?](https://mlsanity.com/paper/1906.00189/) (2019)
- [Learning Deep Kernels for Non-Parametric Two-Sample Tests](https://mlsanity.com/paper/2002.09116/) (2020)
- [Learning with Symmetric Label Noise: The Importance of Being Unhinged](https://mlsanity.com/paper/1505.07634/) (2015)
- [To Smooth or Not? When Label Smoothing Meets Noisy Labels](https://mlsanity.com/paper/2106.04149/) (2021)
- [Interpretable Distribution Features with Maximum Testing Power](https://mlsanity.com/paper/1605.06796/) (2016)
- [On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means](https://mlsanity.com/paper/1904.04017/) (2019)
- [Fast Two-Sample Testing with Analytic Representations of Probability Measures](https://mlsanity.com/paper/1506.04725/) (2015)
- [Regressive Domain Adaptation for Unsupervised Keypoint Detection](https://mlsanity.com/paper/2103.06175/) (2021)
- [Robust Learning from Untrusted Sources](https://mlsanity.com/paper/1901.10310/) (2019)
- [Two-sample Testing Using Deep Learning](https://mlsanity.com/paper/1910.06239/) (2019)
- [Optimal Bounds between f-Divergences and Integral Probability Metrics](https://mlsanity.com/paper/2006.05973/) (2020)

## Abstract (arXiv)

Real-life data are often non-IID due to complex distributions and interactions, and the sensitivity to the distribution of samples can differ among learning models. Accordingly, a key question for any supervised or unsupervised model is whether the probability distributions of two given datasets can be considered identical. To address this question, we introduce R-divergence, designed to assess model-oriented distribution discrepancies. The core insight is that two distributions are likely identical if their optimal hypothesis yields the same expected risk for each distribution. To estimate the distribution discrepancy between two datasets, R-divergence learns a minimum hypothesis on the mixed data and then gauges the empirical risk difference between them. We evaluate the test power across various unsupervised and supervised tasks and find that R-divergence achieves state-of-the-art performance. To demonstrate the practicality of R-divergence, we employ R-divergence to train robust neural networks on samples with noisy labels.