# Featurized Density Ratio Estimation

arXiv: https://arxiv.org/abs/2107.02212 · HTML: https://mlsanity.com/paper/2107.02212/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 5 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)
- [MADE: Masked Autoencoder for Distribution Estimation](https://mlsanity.com/paper/1502.03509/) (2015)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)
- [Learning Likelihoods with Conditional Normalizing Flows](https://mlsanity.com/paper/1912.00042/) (2019)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [i-RevNet: Deep Invertible Networks](https://mlsanity.com/paper/1802.07088/) (2018)
- [Fair Generative Modeling via Weak Supervision](https://mlsanity.com/paper/1910.12008/) (2019)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Likelihood-free inference by ratio estimation](https://mlsanity.com/paper/1611.10242/) (2016)
- [Auxiliary Deep Generative Models](https://mlsanity.com/paper/1602.05473/) (2016)
- [Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting](https://mlsanity.com/paper/1906.09531/) (2019)
- [Hybrid Models with Deep and Invertible Features](https://mlsanity.com/paper/1902.02767/) (2019)
- [MintNet: Building Invertible Neural Networks with Masked Convolutions](https://mlsanity.com/paper/1907.07945/) (2019)
- [A RAD approach to deep mixture models](https://mlsanity.com/paper/1903.07714/) (2019)
- [Nearest Neighbor-based Importance Weighting](https://mlsanity.com/paper/2102.02291/) (2021)
- [Constructive Setting of the Density Ratio Estimation Problem and its Rigorous Solution](https://mlsanity.com/paper/1306.0407/) (2013)
- [Bayesian Semisupervised Learning with Deep Generative Models](https://mlsanity.com/paper/1706.09751/) (2017)

## Built on by (most cited first, 30 of 33)

- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [Density Ratio Estimation via Infinitesimal Classification](https://mlsanity.com/paper/2111.11010/) (2021)
- [Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regression](https://mlsanity.com/paper/2305.00869/) (2023)
- [A Short Survey on Importance Weighting for Machine Learning](https://mlsanity.com/paper/2403.10175/) (2024)
- [Dequantified Diffusion-Schr"odinger Bridge for Density Ratio Estimation](https://mlsanity.com/paper/2505.05034/) (2025)
- [Incentives in Private Collaborative Machine Learning](https://mlsanity.com/paper/2404.01676/) (2024)
- [Preference Optimization by Estimating the Ratio of the Data Distribution](https://mlsanity.com/paper/2505.19601/) (2025)
- [Computing high-dimensional optimal transport by flow neural networks](https://mlsanity.com/paper/2305.11857/) (2023)
- [Density Ratio Estimation via Sampling along Generalized Geodesics on Statistical Manifolds](https://mlsanity.com/paper/2406.18806/) (2024)
- [Your copula is a classifier in disguise: classification-based copula density estimation](https://mlsanity.com/paper/2411.03014/) (2024)
- [Diffusion and Flow-based Copulas: Forgetting and Remembering Dependencies](https://mlsanity.com/paper/2509.19707/) (2025)
- [A Unified Framework for Multi-distribution Density Ratio Estimation](https://mlsanity.com/paper/2112.03440/) (2021)
- [Overcoming Saturation in Density Ratio Estimation by Iterated Regularization](https://mlsanity.com/paper/2402.13891/) (2024)
- [Projection Pursuit Density Ratio Estimation](https://mlsanity.com/paper/2506.00866/) (2025)
- [Approximate Data Deletion in Generative Models](https://mlsanity.com/paper/2206.14439/) (2022)
- [General Frameworks for Conditional Two-Sample Testing](https://mlsanity.com/paper/2410.16636/) (2024)
- [Expected information gain estimation via density approximations: Sample allocation and dimension reduction](https://mlsanity.com/paper/2411.08390/) (2024)
- [Diffusion Secant Alignment for Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2509.04852/) (2025)
- [A Minimum Variance Path Principle for Accurate and Stable Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2602.00834/) (2026)
- [Off-Policy Evaluation in Embedded Spaces](https://mlsanity.com/paper/2203.02807/) (2022)
- [Density Ratio Estimation and Neyman Pearson Classification with Missing Data](https://mlsanity.com/paper/2302.10655/) (2023)
- [Binary Losses for Density Ratio Estimation](https://mlsanity.com/paper/2407.01371/) (2024)
- [On the Difficulty of Learning a Meta-network for Training Data Selection](https://mlsanity.com/paper/2606.00571/) (2026)
- [Enhanced Importance Sampling through Latent Space Exploration in Normalizing Flows](https://mlsanity.com/paper/2501.03394/) (2025)
- [Two-sample comparison through additive tree models for density ratios](https://mlsanity.com/paper/2508.03059/) (2025)
- [One-Step Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2604.10672/) (2026)
- [Unified Perspective on Probability Divergence via Maximum Likelihood Density Ratio Estimation: Bridging KL-Divergence and Integral Probability Metrics](https://mlsanity.com/paper/2201.13127/) (2022)
- [Mutual Information Multinomial Estimation](https://mlsanity.com/paper/2408.09377/) (2024)
- [A density ratio framework for evaluating the utility of synthetic data](https://mlsanity.com/paper/2408.13167/) (2024)
- [Neural Total Variation Distance Estimators for Changepoint Detection in News Data](https://mlsanity.com/paper/2506.18764/) (2025)

## Abstract (arXiv)

Density ratio estimation serves as an important technique in the unsupervised machine learning toolbox. However, such ratios are difficult to estimate for complex, high-dimensional data, particularly when the densities of interest are sufficiently different. In our work, we propose to leverage an invertible generative model to map the two distributions into a common feature space prior to estimation. This featurization brings the densities closer together in latent space, sidestepping pathological scenarios where the learned density ratios in input space can be arbitrarily inaccurate. At the same time, the invertibility of our feature map guarantees that the ratios computed in feature space are equivalent to those in input space. Empirically, we demonstrate the efficacy of our approach in a variety of downstream tasks that require access to accurate density ratios such as mutual information estimation, targeted sampling in deep generative models, and classification with data augmentation.