# A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity

arXiv: https://arxiv.org/abs/2509.24734 · HTML: https://mlsanity.com/paper/2509.24734/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 40 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023)
- [VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset](https://mlsanity.com/paper/2305.18500/) (2023)
- [mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video](https://mlsanity.com/paper/2302.00402/) (2023)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [OmniVL:One Foundation Model for Image-Language and Video-Language Tasks](https://mlsanity.com/paper/2209.07526/) (2022)
- [VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset](https://mlsanity.com/paper/2304.08345/) (2023)
- [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://mlsanity.com/paper/2402.13217/) (2024)
- [Learning Audio-Video Modalities from Image Captions](https://mlsanity.com/paper/2204.00679/) (2022)
- [AudioMNIST: Exploring Explainable Artificial Intelligence for Audio Analysis on a Simple Benchmark](https://mlsanity.com/paper/1807.03418/) (2018)
- [HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training](https://mlsanity.com/paper/2212.14546/) (2022)
- [VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners](https://mlsanity.com/paper/2212.04979/) (2022)
- [A Touch, Vision, and Language Dataset for Multimodal Alignment](https://mlsanity.com/paper/2402.13232/) (2024)
- [Gramian Multimodal Representation Learning and Alignment](https://mlsanity.com/paper/2412.11959/) (2024)
- [Multi-granularity Correspondence Learning from Long-term Noisy Videos](https://mlsanity.com/paper/2401.16702/) (2024)
- [Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities](https://mlsanity.com/paper/2411.01053/) (2024)
- [Accommodating Audio Modality in CLIP for Multimodal Processing](https://mlsanity.com/paper/2303.06591/) (2023)

## Built on by (most cited first, 18 of 18)

- [The More, the Merrier: Contrastive Fusion for Higher-Order Multimodal Alignment](https://mlsanity.com/paper/2511.21331/) (2025)
- [Multi-Way Representation Alignment](https://mlsanity.com/paper/2602.06205/) (2026)
- [DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation](https://mlsanity.com/paper/2605.30350/) (2026)
- [Calibrated Multimodal Representation Learning with Missing Modalities](https://mlsanity.com/paper/2511.12034/) (2025)
- [Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning](https://mlsanity.com/paper/2604.05834/) (2026)
- [Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning](https://mlsanity.com/paper/2607.17673/) (2026)
- [Binding Multiple Modalities via Multimodal Wasserstein Barycenter](https://mlsanity.com/paper/2609.33800/) (2026)
- [Toward Unified Multimodal Representation Learning for Autonomous Driving](https://mlsanity.com/paper/2603.07874/) (2026)
- [RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval?](https://mlsanity.com/paper/2605.27436/) (2026)
- [OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment](https://mlsanity.com/paper/2605.29900/) (2026)
- [GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention](https://mlsanity.com/paper/2606.06249/) (2026)
- [Trimodal Glioma Representation Alignment via Volumetric Contrastive Learning](https://mlsanity.com/paper/2606.14568/) (2026)
- [What Images Cannot Say: Language-Guided Olfactory Representation Learning](https://mlsanity.com/paper/2607.06402/) (2026)
- [Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval](https://mlsanity.com/paper/2609.15320/) (2026)
- [TRIM-ReID: Duplication-Aware Token Reduction and Modality-Aligned Interaction for Multi-Modal Object Re-Identification](https://mlsanity.com/paper/2610.04361/) (2026)
- [HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling](https://mlsanity.com/paper/2610.12393/) (2026)
- [Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning](https://mlsanity.com/paper/2610.14770/) (2026)
- [Omni-Modal LLMs Know More Than They Say](https://mlsanity.com/paper/2610.16135/) (2026)

## Abstract (arXiv)

Multimodal learning plays a pivotal role in advancing artificial intelligence systems by incorporating information from multiple modalities to build a more comprehensive representation. Despite its importance, current state-of-the-art models still suffer from severe limitations that prevent the successful development of a fully multimodal model. Such methods may not provide indicators that all the involved modalities are effectively aligned. As a result, some modalities may not be aligned, undermining the effectiveness of the model in downstream tasks where multiple modalities should provide additional information that the model fails to exploit. In this paper, we present TRIANGLE: TRI-modAl Neural Geometric LEarning, the novel proposed similarity measure that is directly computed in the higher-dimensional space spanned by the modality embeddings. TRIANGLE improves the joint alignment of three modalities via a triangle-area similarity, avoiding additional fusion layers or pairwise similarities. When incorporated in contrastive losses replacing cosine similarity, TRIANGLE significantly boosts the performance of multimodal modeling, while yielding interpretable alignment rationales. Extensive evaluation in three-modal tasks such as video-text and audio-text retrieval or audio-video classification, demonstrates that TRIANGLE achieves state-of-the-art results across different datasets improving the performance of cosine-based methods up to 9 points of Recall@1.