# Overcoming Saturation in Density Ratio Estimation by Iterated Regularization

arXiv: https://arxiv.org/abs/2402.13891 · HTML: https://mlsanity.com/paper/2402.13891/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 98 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Positive-Unlabeled Learning with Non-Negative Risk Estimator](https://mlsanity.com/paper/1703.00593/) (2017)
- [A DIRT-T Approach to Unsupervised Domain Adaptation](https://mlsanity.com/paper/1802.08735/) (2018)
- [Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning](https://mlsanity.com/paper/1702.08811/) (2017)
- [Generalization Properties of Learning with Random Features](https://mlsanity.com/paper/1602.04474/) (2016)
- [Correlation Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1612.01939/) (2016)
- [Learning in Implicit Generative Models](https://mlsanity.com/paper/1610.03483/) (2016)
- [Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks](https://mlsanity.com/paper/1904.13262/) (2019)
- [Deep Subdomain Adaptation Network for Image Classification](https://mlsanity.com/paper/2106.09388/) (2021)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models](https://mlsanity.com/paper/2211.17091/) (2022)
- [HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1912.11976/) (2019)
- [Multi-Source Deep Domain Adaptation with Weak Supervision for Time-Series Sensor Data](https://mlsanity.com/paper/2005.10996/) (2020)
- [Optimal Rates For Regularization Of Statistical Inverse Learning Problems](https://mlsanity.com/paper/1604.04054/) (2016)
- [Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spaces](https://mlsanity.com/paper/1801.06720/) (2018)
- [Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation](https://mlsanity.com/paper/2006.06979/) (2020)
- [ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data](https://mlsanity.com/paper/2203.08321/) (2022)
- [A General Framework for Consistent Structured Prediction with Implicit Loss Embeddings](https://mlsanity.com/paper/2002.05424/) (2020)
- [On Minimum Discrepancy Estimation for Deep Domain Adaptation](https://mlsanity.com/paper/1901.00282/) (2019)
- [Featurized Density Ratio Estimation](https://mlsanity.com/paper/2107.02212/) (2021)
- [Finite-sample analysis of M-estimators using self-concordance](https://mlsanity.com/paper/1810.06838/) (2018)
- [Beyond Least-Squares: Fast Rates for Regularized Empirical Risk Minimization through Self-Concordance](https://mlsanity.com/paper/1902.03046/) (2019)
- [Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses](https://mlsanity.com/paper/1907.01771/) (2019)
- [Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regression](https://mlsanity.com/paper/2305.00869/) (2023)
- [Addressing Parameter Choice Issues in Unsupervised Domain Adaptation by Aggregation](https://mlsanity.com/paper/2305.01281/) (2023)
- [Kernel Conditional Density Operators](https://mlsanity.com/paper/1905.11255/) (2019)

## Built on by (most cited first, 5 of 5)

- [Projection Pursuit Density Ratio Estimation](https://mlsanity.com/paper/2506.00866/) (2025)
- [Diffusion Secant Alignment for Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2509.04852/) (2025)
- [Binary Losses for Density Ratio Estimation](https://mlsanity.com/paper/2407.01371/) (2024)
- [One-Step Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2604.10672/) (2026)
- [Mutual Information Multinomial Estimation](https://mlsanity.com/paper/2408.09377/) (2024)

## Abstract (arXiv)

Estimating the ratio of two probability densities from finitely many samples, is a central task in machine learning and statistics. In this work, we show that a large class of kernel methods for density ratio estimation suffers from error saturation, which prevents algorithms from achieving fast error convergence rates on highly regular learning problems. To resolve saturation, we introduce iterated regularization in density ratio estimation to achieve fast error rates. Our methods outperform its non-iteratively regularized versions on benchmarks for density ratio estimation as well as on large-scale evaluations for importance-weighted ensembling of deep unsupervised domain adaptation models.