# Spectral Discovery of Jointly Smooth Features for Multimodal Data

arXiv: https://arxiv.org/abs/2004.04386 · HTML: https://mlsanity.com/paper/2004.04386/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [On the Parameter Combinations That Matter and on Those That do Not](https://mlsanity.com/paper/2110.06717/) (2021)
- [A common variable minimax theorem for graphs](https://mlsanity.com/paper/2107.14747/) (2021)
- [On Learning what to Learn: heterogeneous observations of dynamics and establishing (possibly causal) relations among them](https://mlsanity.com/paper/2406.06812/) (2024)
- [Nonlinear Manifold Learning Determines Microgel Size from Raman Spectroscopy](https://mlsanity.com/paper/2403.08376/) (2024)
- [Complex Interpolation of Matrices with an application to Multi-Manifold Learning](https://mlsanity.com/paper/2604.14118/) (2026)

## Abstract (arXiv)

In this paper, we propose a spectral method for deriving functions that are jointly smooth on multiple observed manifolds. This allows us to register measurements of the same phenomenon by heterogeneous sensors, and to reject sensor-specific noise. Our method is unsupervised and primarily consists of two steps. First, using kernels, we obtain a subspace spanning smooth functions on each separate manifold. Then, we apply a spectral method to the obtained subspaces and discover functions that are jointly smooth on all manifolds. We show analytically that our method is guaranteed to provide a set of orthogonal functions that are as jointly smooth as possible, ordered by increasing Dirichlet energy from the smoothest to the least smooth. In addition, we show that the extracted functions can be efficiently extended to unseen data using the Nystr"om method. We demonstrate the proposed method on both simulated and real measured data and compare the results to nonlinear variants of the seminal Canonical Correlation Analysis (CCA). Particularly, we show superior results for sleep stage identification. In addition, we show how the proposed method can be leveraged for finding minimal realizations of parameter spaces of nonlinear dynamical systems.