# Relative gradient optimization of the Jacobian term in unsupervised deep learning

arXiv: https://arxiv.org/abs/2006.15090 · HTML: https://mlsanity.com/paper/2006.15090/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2020.

## Built on by (most cited first, 16 of 16)

- [Nonparametric Identifiability of Causal Representations from Unknown Interventions](https://mlsanity.com/paper/2306.00542/) (2023)
- [Independent mechanism analysis, a new concept?](https://mlsanity.com/paper/2106.05200/) (2021)
- [Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA](https://mlsanity.com/paper/2106.09620/) (2021)
- [Identifiability of latent-variable and structural-equation models: from linear to nonlinear](https://mlsanity.com/paper/2302.02672/) (2023)
- [Causal Component Analysis](https://mlsanity.com/paper/2305.17225/) (2023)
- [Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning](https://mlsanity.com/paper/2303.16535/) (2023)
- [CoCo: A Coupled Contrastive Framework for Unsupervised Domain Adaptive Graph Classification](https://mlsanity.com/paper/2306.04979/) (2023)
- [Embrace the Gap: VAEs Perform Independent Mechanism Analysis](https://mlsanity.com/paper/2206.02416/) (2022)
- [Free-form Flows: Make Any Architecture a Normalizing Flow](https://mlsanity.com/paper/2310.16624/) (2023)
- [Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow](https://mlsanity.com/paper/2405.13629/) (2024)
- [Lifting Architectural Constraints of Injective Flows](https://mlsanity.com/paper/2306.01843/) (2023)
- [A Review of Change of Variable Formulas for Generative Modeling](https://mlsanity.com/paper/2308.02652/) (2023)
- [Identifiable Causal Representation Learning: Unsupervised, Multi-View, and Multi-Environment](https://mlsanity.com/paper/2406.13371/) (2024)
- [Training Energy-Based Normalizing Flow with Score-Matching Objectives](https://mlsanity.com/paper/2305.15267/) (2023)
- [Probing the Robustness of Independent Mechanism Analysis for Representation Learning](https://mlsanity.com/paper/2207.06137/) (2022)
- [Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiability](https://mlsanity.com/paper/2510.17040/) (2025)

## Abstract (arXiv)

Learning expressive probabilistic models correctly describing the data is a ubiquitous problem in machine learning. A popular approach for solving it is mapping the observations into a representation space with a simple joint distribution, which can typically be written as a product of its marginals -- thus drawing a connection with the field of nonlinear independent component analysis. Deep density models have been widely used for this task, but their maximum likelihood based training requires estimating the log-determinant of the Jacobian and is computationally expensive, thus imposing a trade-off between computation and expressive power. In this work, we propose a new approach for exact training of such neural networks. Based on relative gradients, we exploit the matrix structure of neural network parameters to compute updates efficiently even in high-dimensional spaces; the computational cost of the training is quadratic in the input size, in contrast with the cubic scaling of naive approaches. This allows fast training with objective functions involving the log-determinant of the Jacobian, without imposing constraints on its structure, in stark contrast to autoregressive normalizing flows.