# Unsupervised learning of disentangled representations in deep restricted kernel machines with orthogonality constraints

arXiv: https://arxiv.org/abs/2011.12659 · HTML: https://mlsanity.com/paper/2011.12659/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2020.

## Built on by (most cited first, 9 of 9)

- [TRKM: Twin Restricted Kernel Machines for Classification and Regression](https://mlsanity.com/paper/2502.15759/) (2025)
- [Deep Kernel Principal Component Analysis for Multi-level Feature Learning](https://mlsanity.com/paper/2302.11220/) (2023)
- [Tensor-based Multi-view Spectral Clustering via Shared Latent Space](https://mlsanity.com/paper/2207.11559/) (2022)
- [Unsupervised Neighborhood Propagation Kernel Layers for Semi-supervised Node Classification](https://mlsanity.com/paper/2301.13764/) (2023)
- [Randomized based restricted kernel machine for hyperspectral image classification](https://mlsanity.com/paper/2503.05837/) (2025)
- [Multi-view Kernel PCA for Time series Forecasting](https://mlsanity.com/paper/2301.09811/) (2023)
- [Duality in Multi-View Restricted Kernel Machines](https://mlsanity.com/paper/2305.17251/) (2023)
- [Combining Primal and Dual Representations in Deep Restricted Kernel Machines Classifiers](https://mlsanity.com/paper/2306.07015/) (2023)
- [CI-RKM: A Class-Informed Approach to Robust Restricted Kernel Machines](https://mlsanity.com/paper/2504.11476/) (2025)

## Abstract (arXiv)

We introduce Constr-DRKM, a deep kernel method for the unsupervised learning of disentangled data representations. We propose augmenting the original deep restricted kernel machine formulation for kernel PCA by orthogonality constraints on the latent variables to promote disentanglement and to make it possible to carry out optimization without first defining a stabilized objective. After illustrating an end-to-end training procedure based on a quadratic penalty optimization algorithm with warm start, we quantitatively evaluate the proposed method's effectiveness in disentangled feature learning. We demonstrate on four benchmark datasets that this approach performs similarly overall to β-VAE on a number of disentanglement metrics when few training points are available, while being less sensitive to randomness and hyperparameter selection than β-VAE. We also present a deterministic initialization of Constr-DRKM's training algorithm that significantly improves the reproducibility of the results. Finally, we empirically evaluate and discuss the role of the number of layers in the proposed methodology, examining the influence of each principal component in every layer and showing that components in lower layers act as local feature detectors capturing the broad trends of the data distribution, while components in deeper layers use the representation learned by previous layers and more accurately reproduce higher-level features.