# Neural Methods for Point-wise Dependency Estimation

arXiv: https://arxiv.org/abs/2006.05553 · HTML: https://mlsanity.com/paper/2006.05553/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 22 of 22)

- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders](https://mlsanity.com/paper/2205.10053/) (2022)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [Contrastive Difference Predictive Coding](https://mlsanity.com/paper/2310.20141/) (2023)
- [Self-supervised Representation Learning with Relative Predictive Coding](https://mlsanity.com/paper/2103.11275/) (2021)
- [Approximating mutual information of high-dimensional variables using learned representations](https://mlsanity.com/paper/2409.02732/) (2024)
- [Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planning](https://mlsanity.com/paper/2402.00251/) (2024)
- [A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2206.04551/) (2022)
- [A Novel Approach for Effective Multi-View Clustering with Information-Theoretic Perspective](https://mlsanity.com/paper/2309.13989/) (2023)
- [Neural Conditional Probability for Uncertainty Quantification](https://mlsanity.com/paper/2407.01171/) (2024)
- [Content-Based Medical Image Retrieval with Opponent Class Adaptive Margin Loss](https://mlsanity.com/paper/2211.15371/) (2022)
- [Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis](https://mlsanity.com/paper/2304.00395/) (2023)
- [Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery](https://mlsanity.com/paper/2204.13906/) (2022)
- [Efficient Parametric SVD of Koopman Operator for Stochastic Dynamical Systems](https://mlsanity.com/paper/2507.07222/) (2025)
- [Demystifying Spectral Feature Learning for Instrumental Variable Regression](https://mlsanity.com/paper/2506.10899/) (2025)
- [Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning](https://mlsanity.com/paper/2510.20644/) (2025)
- [Contrastive Predictive Coding Done Right for Mutual Information Estimation](https://mlsanity.com/paper/2510.25983/) (2025)
- [Cross-View Graph Consistency Learning for Invariant Graph Representations](https://mlsanity.com/paper/2311.11821/) (2023)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate](https://mlsanity.com/paper/2606.00241/) (2026)
- [Weakly Supervised Recovery of Semantic Attributes](https://mlsanity.com/paper/2103.11888/) (2021)
- [Neural PCA for Flow-Based Representation Learning](https://mlsanity.com/paper/2208.10753/) (2022)

## Abstract (arXiv)

Since its inception, the neural estimation of mutual information (MI) has demonstrated the empirical success of modeling expected dependency between high-dimensional random variables. However, MI is an aggregate statistic and cannot be used to measure point-wise dependency between different events. In this work, instead of estimating the expected dependency, we focus on estimating point-wise dependency (PD), which quantitatively measures how likely two outcomes co-occur. We show that we can naturally obtain PD when we are optimizing MI neural variational bounds. However, optimizing these bounds is challenging due to its large variance in practice. To address this issue, we develop two methods (free of optimizing MI variational bounds): Probabilistic Classifier and Density-Ratio Fitting. We demonstrate the effectiveness of our approaches in 1) MI estimation, 2) self-supervised representation learning, and 3) cross-modal retrieval task.