mlsanityData to 7 Oct 2026

Gacs-Korner Common Information Variational Autoencoder

arXiv 2205.12239 ↗First posted 24 May 2022cs.LG cs.CV cs.ITcited by 12

Not on the map. Why?

Abstract (arXiv)

We propose a notion of common information that allows one to quantify and separate the information that is shared between two random variables from the information that is unique to each. Our notion of common information is defined by an optimization problem over a family of functions and recovers the G'acs-K"orner common information as a special case. Importantly, our notion can be approximated empirically using samples from the underlying data distribution. We then provide a method to partition and quantify the common and unique information using a simple modification of a traditional variational auto-encoder. Empirically, we demonstrate that our formulation allows us to learn semantically meaningful common and unique factors of variation even on high-dimensional data such as images and videos. Moreover, on datasets where ground-truth latent factors are known, we show that we can accurately quantify the common information between the random variables.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 27 of its 50 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

27 of 50 references.

References · 27

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2019Contrastive Multiview Codingcited by 1,297
  4. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  5. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  6. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  7. 2019On Variational Bounds of Mutual Informationcited by 550
  8. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  9. 2018Understanding disentangling in β-VAEcited by 332
  10. 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
  11. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  12. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
15 more
  1. 2016On Deep Multi-View Representation Learning: Objectives and Optimizationcited by 162
  2. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  3. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  4. 2018Disentangled Sequential Autoencodercited by 117
  5. 2017Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observationscited by 100
  6. 2019Practical Lossless Compression with Latent Variables using Bits Back Codingcited by 85
  7. 2022Rethinking Minimal Sufficient Representation in Contrastive Learningcited by 67
  8. 2019Learning Disentangled Representations via Mutual Information Estimationcited by 52
  9. 2018Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoderscited by 49
  10. 2016Deep Variational Canonical Correlation Analysiscited by 44
  11. 2020Learning Optimal Representations with the Decodable Information Bottleneckcited by 32
  12. 2019A Novel Approach to the Partial Information Decompositioncited by 16
  13. 2020NestedVAE: Isolating Common Factors via Weak Supervisioncited by 13
  14. 2018Estimating Information Flow in Deep Neural Networkscited by 7
  15. 2020Usable Information and Evolution of Optimal Representations During Trainingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 5; 2026: 2.

12 papers.

  1. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  2. 2022Critical Learning Periods for Multisensory Integration in Deep Networkscited by 14
  3. 2024Minimum Description Length and Generalization Guarantees for Representation Learningcited by 11
  4. 2023Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Datacited by 9
  5. 2024Discovering Common Information in Multi-view Datacited by 3
  6. 2025InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signalscited by 3
  7. 2025Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoderscited by 3
  8. 2025Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priorscited by 2
  9. 2025Multiview Representation Learning via Distributed Joint Latent Space Structuringcited by 1
  10. 2025Towards Comprehensive Information-theoretic Multi-view Learningcited by 1
  11. 2026Diff-CA: Separating Common and Salient Factors with Diffusion Modelscited by 1
  12. 2026Counterfactual Contrastive Analysiscited by 0