# Hierarchical Contrastive Learning for Multimodal Data

arXiv: https://arxiv.org/abs/2604.05462 · HTML: https://mlsanity.com/paper/2604.05462/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Builds on (most cited first, 11 of 11)

- [Multi-View Causal Representation Learning with Partial Observability](https://mlsanity.com/paper/2311.04056/) (2023)
- [What to align in multimodal contrastive learning?](https://mlsanity.com/paper/2409.07402/) (2024)
- [On the Value of Cross-Modal Misalignment in Multimodal Representation Learning](https://mlsanity.com/paper/2504.10143/) (2025)
- [Estimating shared subspace with AJIVE: the power and limitation of multiple data matrices](https://mlsanity.com/paper/2501.09336/) (2025)
- [Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2602.19585/) (2026)
- [A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics](https://mlsanity.com/paper/2503.17538/) (2025)
- [Contrastive Learning on Multimodal Analysis of Electronic Health Records](https://mlsanity.com/paper/2403.14926/) (2024)
- [Spectral Joint Subspace Estimation for Heterogeneous Multi-View Data: Geometry and Reweighting](https://mlsanity.com/paper/2512.02866/) (2025)
- [Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations](https://mlsanity.com/paper/2602.14983/) (2026)
- [SPANER: Shared Prompt Aligner for Multimodal Semantic Representation](https://mlsanity.com/paper/2508.13387/) (2025)
- [Revisiting Theory of Contrastive Learning for Domain Generalization](https://mlsanity.com/paper/2512.02831/) (2025)

## Abstract (arXiv)

Multimodal representation learning is commonly built on a shared-private decomposition, treating latent information as either common to all modalities or specific to one. This binary view is often inadequate: many factors are shared by only subsets of modalities, and ignoring such partial sharing can over-align unrelated signals and obscure complementary information. We propose Hierarchical Contrastive Learning (HCL), a framework that learns globally shared, partially shared, and modality-specific representations within a unified model. HCL combines a hierarchical latent-variable formulation with structural sparsity and a structure-aware contrastive objective that aligns only modalities that genuinely share a latent factor. Under uncorrelated latent variables, we prove identifiability of the hierarchical decomposition, establish recovery guarantees for the loading matrices, and derive parameter estimation and excess-risk bounds for downstream prediction. Simulations show accurate recovery of hierarchical structure and effective selection of task-relevant components. On multimodal electronic health records, HCL yields more informative representations and consistently improves predictive performance.