# Compositional Entailment Learning for Hyperbolic Vision-Language Models

arXiv: https://arxiv.org/abs/2410.06912 · HTML: https://mlsanity.com/paper/2410.06912/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 53 of its 78 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/1910.12892/) (2019)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [Riemannian Adaptive Optimization Methods](https://mlsanity.com/paper/1810.00760/) (2018)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)
- [Poincar'e GloVe: Hyperbolic Word Embeddings](https://mlsanity.com/paper/1810.06546/) (2018)
- [Hyperbolic Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.04562/) (2021)
- [Hyperbolic Image-Text Representations](https://mlsanity.com/paper/2304.09172/) (2023)

## Built on by (most cited first, 30 of 74)

- [HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts](https://mlsanity.com/paper/2505.24722/) (2025)
- [Is CLIP ideal? No. Can we fix it? Yes!](https://mlsanity.com/paper/2503.08723/) (2025)
- [Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries](https://mlsanity.com/paper/2504.08896/) (2025)
- [Hyperbolic Safety-Aware Vision-Language Models](https://mlsanity.com/paper/2503.12127/) (2025)
- [HLFormer: Enhancing Partially Relevant Video Retrieval with Hyperbolic Learning](https://mlsanity.com/paper/2507.17402/) (2025)
- [Hyperbolic Dataset Distillation](https://mlsanity.com/paper/2505.24623/) (2025)
- [Learning Visual Hierarchies in Hyperbolic Space for Image Retrieval](https://mlsanity.com/paper/2411.17490/) (2024)
- [The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition](https://mlsanity.com/paper/2505.24840/) (2025)
- [PHyCLIP: ℓ1-Product of Hyperbolic Factors Unifies Hierarchy and Compositionality in Vision-Language Representation Learning](https://mlsanity.com/paper/2510.08919/) (2025)
- [GeoWorld: Geometric World Models](https://mlsanity.com/paper/2602.23058/) (2026)
- [HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models](https://mlsanity.com/paper/2510.20322/) (2025)
- [Hyperbolic Busemann Neural Networks](https://mlsanity.com/paper/2602.18858/) (2026)
- [HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment](https://mlsanity.com/paper/2603.22721/) (2026)
- [Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learning](https://mlsanity.com/paper/2506.05826/) (2025)
- [Path-Decoupled Hyperbolic Flow Matching for Few-Shot Adaptation](https://mlsanity.com/paper/2602.20479/) (2026)
- [Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models](https://mlsanity.com/paper/2603.00431/) (2026)
- [ProbRes: Probabilistic Jump Diffusion for Open-World Egocentric Activity Recognition](https://mlsanity.com/paper/2504.03948/) (2025)
- [Global and Local Entailment Learning for Natural World Imagery](https://mlsanity.com/paper/2506.21476/) (2025)
- [Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds](https://mlsanity.com/paper/2510.27391/) (2025)
- [NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimization](https://mlsanity.com/paper/2511.08417/) (2025)
- [Understanding and Improving Hyperbolic Deep Reinforcement Learning](https://mlsanity.com/paper/2512.14202/) (2025)
- [Intrinsic Lorentz Neural Network](https://mlsanity.com/paper/2602.23981/) (2026)
- [HyperPath: Knowledge-Guided Hyperbolic Semantic Hierarchy Modeling for WSI Analysis](https://mlsanity.com/paper/2506.16398/) (2025)
- [Learning Protein-Ligand Binding in Hyperbolic Space](https://mlsanity.com/paper/2508.15480/) (2025)
- [Angular Gradient Sign Method: Uncovering Vulnerabilities in Hyperbolic Networks](https://mlsanity.com/paper/2511.12985/) (2025)
- [Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2603.22042/) (2026)
- [Hyperbolic Multimodal Continual Learning](https://mlsanity.com/paper/2608.09572/) (2026)
- [DELST: Dual Entailment Learning for Hyperbolic Image-Gene Pretraining in Spatial Transcriptomics](https://mlsanity.com/paper/2503.00804/) (2025)
- [Hyperbolic Cycle Alignment for Infrared-Visible Image Fusion](https://mlsanity.com/paper/2507.23508/) (2025)
- [H2em: Learning Hierarchical Hyperbolic Embeddings for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2512.20029/) (2025)

## Abstract (arXiv)

Image-text representation learning forms a cornerstone in vision-language models, where pairs of images and textual descriptions are contrastively aligned in a shared embedding space. Since visual and textual concepts are naturally hierarchical, recent work has shown that hyperbolic space can serve as a high-potential manifold to learn vision-language representation with strong downstream performance. In this work, for the first time we show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-text pairs. We propose Compositional Entailment Learning for hyperbolic vision-language models. The idea is that an image is not only described by a sentence but is itself a composition of multiple object boxes, each with their own textual description. Such information can be obtained freely by extracting nouns from sentences and using openly available localized grounding models. We show how to hierarchically organize images, image boxes, and their textual descriptions through contrastive and entailment-based objectives. Empirical evaluation on a hyperbolic vision-language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP learning, as well as recent hyperbolic alternatives, with better zero-shot and retrieval generalization and clearly stronger hierarchical performance.