Unsupervised Representation Learning from Sparse Transformation Analysis
Not on the map. Why?
Abstract (arXiv)
There is a vast literature on representation learning based on principles such as coding efficiency, statistical independence, causality, controllability, or symmetry. In this paper we propose to learn representations from sequence data by factorizing the transformations of the latent variables into sparse components. Input data are first encoded as distributions of latent activations and subsequently transformed using a probability flow model, before being decoded to predict a future input state. The flow model is decomposed into a number of rotational (divergence-free) vector fields and a number of potential flow (curl-free) fields. Our sparsity prior encourages only a small number of these fields to be active at any instant and infers the speed with which the probability flows along these fields. Training this model is completely unsupervised using a standard variational objective and results in a new form of disentangled representations where the input is not only represented by a combination of independent factors, but also by a combination of independent transformation primitives given by the learned flow fields. When viewing the transformations as symmetries one may interpret this as learning approximately equivariant representations. Empirically we demonstrate that this model achieves state of the art in terms of both data likelihood and unsupervised approximate equivariance errors on datasets composed of sequence transformations.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 78 of its 110 references that are in the map. First posted on arXiv 7 Oct 2024.
Builds on
78 of 110 references.
References · 78
- 2017Attention Is All You Needcited by 59,932
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2020Denoising Diffusion Implicit Modelscited by 9,655
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
- 2018Neural Ordinary Differential Equationscited by 3,733
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2015Variational Inference with Normalizing Flowscited by 2,161
- 2016Group Equivariant Convolutional Networkscited by 1,154
- 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
- 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
- 2019Hamiltonian Neural Networkscited by 702
38 more
- 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
- 2018Disentangling by Factorisingcited by 614
- 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
- 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
- 2020Closed-Form Factorization of Latent Semantics in GANscited by 407
- 2022Message Passing Neural PDE Solverscited by 368
- 2022Diffusion Models already have a Semantic Latent Spacecited by 324
- 2016Steerable CNNscited by 298
- 2018Towards a Definition of Disentangled Representationscited by 294
- 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
- 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 288
- 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
- 2020Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Datacited by 248
- 2019On the "steerability" of generative adversarial networkscited by 247
- 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 240
- 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
- 2016Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoderscited by 219
- 2019GANalyze: Toward Visual Definitions of Cognitive Image Propertiescited by 193
- 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
- 2017Equivariance Through Parameter-Sharingcited by 139
- 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
- 2019Hamiltonian Generative Networkscited by 136
- 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 121
- 2023Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometrycited by 121
- 2018Disentangled Sequential Autoencodercited by 117
- 2019Deep Scale-spaces: Equivariance Over Scalecited by 113
- 2019Learning Neural Causal Models from Unknown Interventionscited by 113
- 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
- 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 90
- 2018Learning Disentangled Joint Continuous and Discrete Representationscited by 83
- 2019Learning Neural PDE Solvers with Convergence Guaranteescited by 77
- 2022Neural Conservation Laws: A Divergence-Free Perspectivecited by 67
- 2020S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generationcited by 63
- 2020Guided Variational Autoencoder for Disentanglement Learningcited by 57
- 2021Low-Rank Subspaces in GANscited by 52
- 2021WarpedGANSpace: Finding non-linear RBF paths in GAN latent spacecited by 48
- 2023Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learningcited by 48
- 2023InfoDiffusion: Representation Learning Using Information Maximizing Diffusion Modelscited by 48
50 most cited of 78 All 78 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.