# Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

arXiv: https://arxiv.org/abs/2105.05631 · HTML: https://mlsanity.com/paper/2105.05631/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 73 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 16 of 16)

- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Optimal Transport for Domain Adaptation](https://mlsanity.com/paper/1507.00504/) (2015)
- [A Comprehensive Survey on Cross-modal Retrieval](https://mlsanity.com/paper/1607.06215/) (2016)
- [Learnable Manifold Alignment (LeMA) : A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classification](https://mlsanity.com/paper/1901.02838/) (2019)
- [Shared Generative Latent Representation Learning for Multi-view Clustering](https://mlsanity.com/paper/1907.09747/) (2019)
- [Kernel Manifold Alignment](https://mlsanity.com/paper/1504.02338/) (2015)
- [A Unifying Framework in Vector-valued Reproducing Kernel Hilbert Spaces for Manifold Regularization and Co-Regularized Multi-view Learning](https://mlsanity.com/paper/1401.8066/) (2014)
- [Learning Shared Cross-modality Representation Using Multispectral-LiDAR and Hyperspectral Data](https://mlsanity.com/paper/1912.08837/) (2019)
- [Multi-View Intact Space Learning](https://mlsanity.com/paper/1904.02340/) (2019)
- [CCL: Cross-modal Correlation Learning with Multi-grained Fusion by Hierarchical Network](https://mlsanity.com/paper/1704.02116/) (2017)
- [Selecting the independent coordinates of manifolds with large aspect ratios](https://mlsanity.com/paper/1907.01651/) (2019)
- [SCH-GAN: Semi-supervised Cross-modal Hashing by Generative Adversarial Network](https://mlsanity.com/paper/1802.02488/) (2018)
- [MultiView Diffusion Maps](https://mlsanity.com/paper/1508.05550/) (2015)
- [Weakly-paired Cross-Modal Hashing](https://mlsanity.com/paper/1905.12203/) (2019)
- [Learning Shared Semantic Space with Correlation Alignment for Cross-modal Event Retrieval](https://mlsanity.com/paper/1901.04268/) (2019)
- [Shape Classification using Spectral Graph Wavelets](https://mlsanity.com/paper/1705.06250/) (2017)

## Built on by (most cited first, 3 of 3)

- [Geometric Multimodal Deep Learning with Multi-Scaled Graph Wavelet Convolutional Network](https://mlsanity.com/paper/2111.13361/) (2021)
- [Graph Alignment via Dual-Pass Spectral Encoding and Latent Space Communication](https://mlsanity.com/paper/2509.09597/) (2025)
- [Functional Attention: From Pairwise Affinities to Functional Correspondences](https://mlsanity.com/paper/2605.31559/) (2026)

## Abstract (arXiv)

Multimodal manifold modeling methods extend the spectral geometry-aware data analysis to learning from several related and complementary modalities. Most of these methods work based on two major assumptions: 1) there are the same number of homogeneous data samples in each modality, and 2) at least partial correspondences between modalities are given in advance as prior knowledge. This work proposes two new multimodal modeling methods. The first method establishes a general analyzing framework to deal with the multimodal information problem for heterogeneous data without any specific prior knowledge. For this purpose, first, we identify the localities of each manifold by extracting local descriptors via spectral graph wavelet signatures (SGWS). Then, we propose a manifold regularization framework based on the functional mapping between SGWS descriptors (FMBSD) for finding the pointwise correspondences. The second method is a manifold regularized multimodal classification based on pointwise correspondences (M2CPC) used for the problem of multiclass classification of multimodal heterogeneous, which the correspondences between modalities are determined based on the FMBSD method. The experimental results of evaluating the FMBSD method on three common cross-modal retrieval datasets and evaluating the (M2CPC) method on three benchmark multimodal multiclass classification datasets indicate their effectiveness and superiority over state-of-the-art methods.