# Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data

arXiv: https://arxiv.org/abs/2210.17311 · HTML: https://mlsanity.com/paper/2210.17311/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 55 references that are in the map.
- First posted on arXiv 25 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification](https://mlsanity.com/paper/2008.05457/) (2020)
- [Going Deeper with Contextual CNN for Hyperspectral Image Classification](https://mlsanity.com/paper/1604.03519/) (2016)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model](https://mlsanity.com/paper/2105.10196/) (2021)
- [Classification of Hyperspectral and LiDAR Data Using Coupled CNNs](https://mlsanity.com/paper/2002.01144/) (2020)
- [Learnable Manifold Alignment (LeMA) : A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classification](https://mlsanity.com/paper/1901.02838/) (2019)
- [GaborNet: Gabor filters with learnable parameters in deep convolutional neural networks](https://mlsanity.com/paper/1904.13204/) (2019)
- [Hyperspectral Image Classification with Markov Random Fields and a Convolutional Neural Network](https://mlsanity.com/paper/1705.00727/) (2017)
- [CoSpace: Common Subspace Learning from Hyperspectral-Multispectral Correspondences](https://mlsanity.com/paper/1812.11501/) (2018)
- [Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysis](https://mlsanity.com/paper/1904.08138/) (2019)
- [Invariant Attribute Profiles: A Spatial-Frequency Joint Feature Extractor for Hyperspectral Image Classification](https://mlsanity.com/paper/1912.08847/) (2019)
- [Image Style Transfer and Content-Style Disentanglement](https://mlsanity.com/paper/2111.15624/) (2021)

## Built on by (most cited first, 5 of 5)

- [A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection](https://mlsanity.com/paper/2403.08077/) (2024)
- [DiffCLIP: Few-shot Language-driven Multimodal Classifier](https://mlsanity.com/paper/2412.07119/) (2024)
- [Multi-Task Learning with Multi-Annotation Triplet Loss for Improved Object Detection](https://mlsanity.com/paper/2504.08054/) (2025)
- [Task-Guided Multi-Annotation Triplet Learning for Remote Sensing Representations](https://mlsanity.com/paper/2604.03837/) (2026)
- [Beyond Task-Driven Features for Object Detection](https://mlsanity.com/paper/2604.03839/) (2026)

## Abstract (arXiv)

Heterogeneous data fusion can enhance the robustness and accuracy of an algorithm on a given task. However, due to the difference in various modalities, aligning the sensors and embedding their information into discriminative and compact representations is challenging. In this paper, we propose a Contrastive learning based MultiModal Alignment Network (CoMMANet) to align data from different sensors into a shared and discriminative manifold where class information is preserved. The proposed architecture uses a multimodal triplet autoencoder to cluster the latent space in such a way that samples of the same classes from each heterogeneous modality are mapped close to each other. Since all the modalities exist in a shared manifold, a unified classification framework is proposed. The resulting latent space representations are fused to perform more robust and accurate classification. In a missing sensor scenario, the latent space of one sensor is easily and efficiently predicted using another sensor's latent space, thereby allowing sensor translation. We conducted extensive experiments on a manually labeled multimodal dataset containing hyperspectral data from AVIRIS-NG and NEON, and LiDAR (light detection and ranging) data from NEON. Lastly, the model is validated on two benchmark datasets: Berlin Dataset (hyperspectral and synthetic aperture radar) and MUUFL Gulfport Dataset (hyperspectral and LiDAR). A comparison made with other methods demonstrates the superiority of this method. We achieved a mean overall accuracy of 94.3% on the MUUFL dataset and the best overall accuracy of 71.26% on the Berlin dataset, which is better than other state-of-the-art approaches.