# Nonlinear SVD with Asymmetric Kernels: feature learning and asymmetric Nystr"om method

arXiv: https://arxiv.org/abs/2306.07040 · HTML: https://mlsanity.com/paper/2306.07040/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 38 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [Kernel methods through the roof: handling billions of points efficiently](https://mlsanity.com/paper/2006.10350/) (2020)
- [Revisiting the Nystrom Method for Improved Large-Scale Machine Learning](https://mlsanity.com/paper/1303.1849/) (2013)
- [Optimality of the Johnson-Lindenstrauss Lemma](https://mlsanity.com/paper/1609.02094/) (2016)
- [On Reproducing Kernel Banach Spaces: Generic Definitions and Unified Framework of Constructions](https://mlsanity.com/paper/1901.01002/) (2019)
- [Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines](https://mlsanity.com/paper/2106.01506/) (2021)
- [Node Representation Learning for Directed Graphs](https://mlsanity.com/paper/1810.09176/) (2018)
- [Learning with Asymmetric Kernels: Least Squares and Feature Interpretation](https://mlsanity.com/paper/2202.01397/) (2022)

## Built on by (most cited first, 5 of 5)

- [Elliptical Attention](https://mlsanity.com/paper/2406.13770/) (2024)
- [Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2402.01476/) (2024)
- [Transformer Meets Twicing: Harnessing Unattended Residual Information](https://mlsanity.com/paper/2503.00687/) (2025)
- [A Kernelizable Primal-Dual Formulation of the Multilinear Singular Value Decomposition](https://mlsanity.com/paper/2410.10504/) (2024)
- [An Attention-based Framework for Fair Contrastive Learning](https://mlsanity.com/paper/2411.14765/) (2024)

## Abstract (arXiv)

Asymmetric data naturally exist in real life, such as directed graphs. Different from the common kernel methods requiring Mercer kernels, this paper tackles the asymmetric kernel-based learning problem. We describe a nonlinear extension of the matrix Singular Value Decomposition through asymmetric kernels, namely KSVD. First, we construct two nonlinear feature mappings w.r.t. rows and columns of the given data matrix. The proposed optimization problem maximizes the variance of each mapping projected onto the subspace spanned by the other, subject to a mutual orthogonality constraint. Through Lagrangian duality, we show that it can be solved by the left and right singular vectors in the feature space induced by the asymmetric kernel. Moreover, we start from the integral equations with a pair of adjoint eigenfunctions corresponding to the singular vectors on an asymmetrical kernel, and extend the Nystr"om method to asymmetric cases through the finite sample approximation, which can be applied to speedup the training in KSVD. Experiments show that asymmetric KSVD learns features outperforming Mercer-kernel based methods that resort to symmetrization, and also verify the effectiveness of the asymmetric Nystr"om method.