# Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regression

arXiv: https://arxiv.org/abs/2305.00869 · HTML: https://mlsanity.com/paper/2305.00869/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 1 May 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning](https://mlsanity.com/paper/1705.07761/) (2017)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)
- [Wasserstein Dependency Measure for Representation Learning](https://mlsanity.com/paper/1903.11780/) (2019)
- [Density Ratio Estimation via Infinitesimal Classification](https://mlsanity.com/paper/2111.11010/) (2021)
- [Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation](https://mlsanity.com/paper/2006.06979/) (2020)
- [Featurized Density Ratio Estimation](https://mlsanity.com/paper/2107.02212/) (2021)
- [Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation](https://mlsanity.com/paper/2110.11271/) (2021)
- [A scaled Bregman theorem with applications](https://mlsanity.com/paper/1607.00360/) (2016)
- [Generative Ratio Matching Networks](https://mlsanity.com/paper/1806.00101/) (2018)

## Built on by (most cited first, 17 of 17)

- [Compositional Foundation Models for Hierarchical Planning](https://mlsanity.com/paper/2309.08587/) (2023)
- [Density Ratio Estimation with Conditional Probability Paths](https://mlsanity.com/paper/2502.02300/) (2025)
- [Constraining Generative Models for Engineering Design with Negative Data](https://mlsanity.com/paper/2306.15166/) (2023)
- [Dequantified Diffusion-Schr"odinger Bridge for Density Ratio Estimation](https://mlsanity.com/paper/2505.05034/) (2025)
- [Your copula is a classifier in disguise: classification-based copula density estimation](https://mlsanity.com/paper/2411.03014/) (2024)
- [Diffusion and Flow-based Copulas: Forgetting and Remembering Dependencies](https://mlsanity.com/paper/2509.19707/) (2025)
- [Overcoming Saturation in Density Ratio Estimation by Iterated Regularization](https://mlsanity.com/paper/2402.13891/) (2024)
- [Beyond the training set: an intuitive method for detecting distribution shift in model-based optimization](https://mlsanity.com/paper/2311.05363/) (2023)
- [Neural Mutual Information Estimation with Vector Copulas](https://mlsanity.com/paper/2510.20968/) (2025)
- [Diffusion Secant Alignment for Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2509.04852/) (2025)
- [Binary Losses for Density Ratio Estimation](https://mlsanity.com/paper/2407.01371/) (2024)
- [Reliable algorithm selection for machine learning-guided design](https://mlsanity.com/paper/2503.20767/) (2025)
- [Learning Energy-Based Models from Stochastic Interpolants using Spatiotemporal Differences](https://mlsanity.com/paper/2605.26850/) (2026)
- [One-Step Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2604.10672/) (2026)
- [Density Ratio Estimation-based Bayesian Optimization with Semi-Supervised Learning](https://mlsanity.com/paper/2305.15612/) (2023)
- [Mutual Information Multinomial Estimation](https://mlsanity.com/paper/2408.09377/) (2024)
- [Robust and Sparse Estimation of Unbounded Density Ratio under Heavy Contamination](https://mlsanity.com/paper/2512.09266/) (2025)

## Abstract (arXiv)

Functions of the ratio of the densities p/q are widely used in machine learning to quantify the discrepancy between the two distributions p and q. For high-dimensional distributions, binary classification-based density ratio estimators have shown great promise. However, when densities are well separated, estimating the density ratio with a binary classifier is challenging. In this work, we show that the state-of-the-art density ratio estimators perform poorly on well-separated cases and demonstrate that this is due to distribution shifts between training and evaluation time. We present an alternative method that leverages multi-class classification for density ratio estimation and does not suffer from distribution shift issues. The method uses a set of auxiliary densities mkk=1K and trains a multi-class logistic regression to classify the samples from p, q, and mkk=1K into K+2 classes. We show that if these auxiliary densities are constructed such that they overlap with p and q, then a multi-class logistic regression allows for estimating log p/q on the domain of any of the K+2 distributions and resolves the distribution shift problems of the current state-of-the-art methods. We compare our method to state-of-the-art density ratio estimators on both synthetic and real datasets and demonstrate its superior performance on the tasks of density ratio estimation, mutual information estimation, and representation learning. Code: https://www.blackswhan.com/mdre/