mlsanityData to 7 Oct 2026

The Representation Jensen-Shannon Divergence

arXiv 2305.16446 ↗First posted 25 May 2023cs.LG cs.IT math.ITcited by 9

Not on the map. Why?

Abstract (arXiv)

Quantifying the difference between probability distributions is crucial in machine learning. However, estimating statistical divergences from empirical samples is challenging due to unknown underlying distributions. This work proposes the representation Jensen-Shannon divergence (RJSD), a novel measure inspired by the traditional Jensen-Shannon divergence. Our approach embeds data into a reproducing kernel Hilbert space (RKHS), representing distributions through uncentered covariance operators. We then compute the Jensen-Shannon divergence between these operators, thereby establishing a proper divergence measure between probability distributions in the input space. We provide estimators based on kernel matrices and empirical covariance matrices using Fourier features. Theoretical analysis reveals that RJSD is a lower bound on the Jensen-Shannon divergence, enabling variational estimation. Additionally, we show that RJSD is a higher-order extension of the maximum mean discrepancy (MMD), providing a more sensitive measure of distributional differences. Our experimental results demonstrate RJSD's superiority in two-sample testing, distribution shift detection, and unsupervised domain adaptation, outperforming state-of-the-art techniques. RJSD's versatility and effectiveness make it a promising tool for machine learning research and applications.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 63 of its 102 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

63 of 102 references.

References · 63

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2017Improved Training of Wasserstein GANscited by 2,223
  6. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  9. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  10. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  12. 2018MINE: Mutual Information Neural Estimationcited by 680
38 more
  1. 2016Deep Transfer Learning with Joint Adaptation Networkscited by 609
  2. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  3. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  4. 2015Deep Kernel Learningcited by 440
  5. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 360
  6. 2016Revisiting Classifier Two-Sample Testscited by 258
  7. 2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 251
  8. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  9. 2017Geometric GANcited by 232
  10. 2016Adversarially Learned Inferencecited by 216
  11. 2016Unrolled Generative Adversarial Networkscited by 208
  12. 2019Generalized Sliced Wasserstein Distancescited by 205
  13. 2017VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learningcited by 199
  14. 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 165
  15. 2020Learning Deep Kernels for Non-Parametric Two-Sample Testscited by 129
  16. 2016Mode Regularized Generative Adversarial Networkscited by 123
  17. 2016R'enyi Divergence Variational Inferencecited by 104
  18. 2017PacGAN: The power of two samples in generative adversarial networkscited by 104
  19. 2016Generative Models and Model Criticism via Optimized Maximum Mean Discrepancycited by 99
  20. 2021Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labelscited by 74
  21. 2021MMD Aggregated Two-Sample Testcited by 74
  22. 2016Interpretable Distribution Features with Maximum Testing Powercited by 63
  23. 2017On the Discrimination-Generalization Tradeoff in GANscited by 48
  24. 2020Maximum Mean Discrepancy Test is Aware of Adversarial Attackscited by 44
  25. 2021Galaxy Zoo DECaLS: Detailed Visual Morphology Measurements from Volunteers and Deep Learning for 314,000 Galaxiescited by 40
  26. 2015Optimal Rates for Random Fourier Featurescited by 38
  27. 2022Information Theory with Kernel Methodscited by 37
  28. 2018Learning deep kernels for exponential family densitiescited by 36
  29. 2019Classification Logit Two-sample Testing by Neural Networkscited by 25
  30. 2021Neural Estimation of Statistical Divergencescited by 25
  31. 2019Prescribed Generative Adversarial Networkscited by 23
  32. 2020GANs May Have No Nash Equilibriacited by 20
  33. 2022Learning fair representation with a parametric integral probability metriccited by 20
  34. 2022A Differential Entropy Estimator for Training Neural Networkscited by 20
  35. 2021Measuring Dependence with Matrix-based Entropy Functionalcited by 19
  36. 2022AutoML Two-Sample Testcited by 19
  37. 2021Learning with Density Matrices and Random Featurescited by 17
  38. 2017Optimal rates of entropy estimation over Lipschitz ballscited by 15

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 3; 2026: 1.

9 papers.

  1. 2023Information Flow in Self-Supervised Learningcited by 16
  2. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 15
  3. 2023Understanding Grokking Through A Robustness Viewpointcited by 10
  4. 2025Visual Instruction Bottleneck Tuningcited by 5
  5. 2024Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformercited by 4
  6. 2023FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learningcited by 2
  7. 2026Batch Normalization Amplifies Memorization and Privacy Riskscited by 1
  8. 2024Towards efficient compression and communication for prototype-based decentralized learningcited by 0
  9. 2025PIPCFR: Pseudo-outcome Imputation with Post-treatment Variables for Individual Treatment Effect Estimationcited by 0