# On Deep Multi-View Representation Learning: Objectives and Optimization

arXiv: https://arxiv.org/abs/1602.01024 · HTML: https://mlsanity.com/paper/1602.01024/

## Key facts

- Cited by 162 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2016.

## Built on by (most cited first, 30 of 162)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Trusted Multi-View Classification](https://mlsanity.com/paper/2102.02051/) (2021)
- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [Deep Clustering: A Comprehensive Survey](https://mlsanity.com/paper/2210.04142/) (2022)
- [Multimodal Representation Learning by Alternating Unimodal Adaptation](https://mlsanity.com/paper/2311.10707/) (2023)
- [To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review](https://mlsanity.com/paper/2304.09355/) (2023)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [Reconsidering Representation Alignment for Multi-view Clustering](https://mlsanity.com/paper/2103.07738/) (2021)
- [Reliable Conflictive Multi-View Learning](https://mlsanity.com/paper/2402.16897/) (2024)
- [This looks more like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagation](https://mlsanity.com/paper/2108.12204/) (2021)
- [A Variational Information Bottleneck Approach to Multi-Omics Data Integration](https://mlsanity.com/paper/2102.03014/) (2021)
- [Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspective](https://mlsanity.com/paper/2106.07115/) (2021)
- [Multimodal Federated Learning on IoT Data](https://mlsanity.com/paper/2109.04833/) (2021)
- [Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model](https://mlsanity.com/paper/2405.18014/) (2024)
- [Understanding the Emergence of Multimodal Representation Alignment](https://mlsanity.com/paper/2502.16282/) (2025)
- [Uncertainty-Aware Multi-View Representation Learning](https://mlsanity.com/paper/2201.05776/) (2022)
- [Decoupled Contrastive Multi-View Clustering with High-Order Random Walks](https://mlsanity.com/paper/2308.11164/) (2023)
- [DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropies](https://mlsanity.com/paper/2301.08164/) (2023)
- [Incomplete Multi-view Clustering via Prototype-based Imputation](https://mlsanity.com/paper/2301.11045/) (2023)
- [Robust and Explainable Autoencoders for Unsupervised Time Series Outlier Detection---Extended Version](https://mlsanity.com/paper/2204.03341/) (2022)
- [Contrastive Multi-view Hyperbolic Hierarchical Clustering](https://mlsanity.com/paper/2205.02618/) (2022)
- [Deep Co-Attention Network for Multi-View Subspace Learning](https://mlsanity.com/paper/2102.07751/) (2021)
- [AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment](https://mlsanity.com/paper/2412.00833/) (2024)
- [Heterogeneous Contrastive Learning](https://mlsanity.com/paper/2105.09401/) (2021)
- [Trusted Unified Feature-Neighborhood Dynamics for Multi-View Classification](https://mlsanity.com/paper/2409.00755/) (2024)
- [Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning](https://mlsanity.com/paper/2210.10041/) (2022)

## Abstract (arXiv)

We consider learning representations (features) in the setting in which we have access to multiple unlabeled views of the data for learning while only one view is available for downstream tasks. Previous work on this problem has proposed several techniques based on deep neural networks, typically involving either autoencoder-like networks with a reconstruction objective or paired feedforward networks with a batch-style correlation-based objective. We analyze several techniques based on prior work, as well as new variants, and compare them empirically on image, speech, and text tasks. We find an advantage for correlation-based representation learning, while the best results on most tasks are obtained with our new variant, deep canonically correlated autoencoders (DCCAE). We also explore a stochastic optimization procedure for minibatch correlation-based objectives and discuss the time/performance trade-offs for kernel-based and neural network-based implementations.