mlsanityData to 14 Oct 2026

Compositional Entailment Learning for Hyperbolic Vision-Language Models

arXiv 2410.06912 ↗First posted 9 Oct 2024cs.CV cs.AI cs.LGcited by 74

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Image-text representation learning forms a cornerstone in vision-language models, where pairs of images and textual descriptions are contrastively aligned in a shared embedding space. Since visual and textual concepts are naturally hierarchical, recent work has shown that hyperbolic space can serve as a high-potential manifold to learn vision-language representation with strong downstream performance. In this work, for the first time we show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-text pairs. We propose Compositional Entailment Learning for hyperbolic vision-language models. The idea is that an image is not only described by a sentence but is itself a composition of multiple object boxes, each with their own textual description. Such information can be obtained freely by extracting nouns from sentences and using openly available localized grounding models. We show how to hierarchically organize images, image boxes, and their textual descriptions through contrastive and entailment-based objectives. Empirical evaluation on a hyperbolic vision-language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP learning, as well as recent hyperbolic alternatives, with better zero-shot and retrieval generalization and clearly stronger hierarchical performance.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 78 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

53 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Hyperbolic Learning with Multimodal Large Language Models9 Aug
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  12. 2021Grounded Language-Image Pre-trainingcited by 1,156
38 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  3. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  4. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 684
  5. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  6. 2018Hyperbolic Neural Networkscited by 402
  7. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  8. 2019Hyperbolic Graph Neural Networkscited by 244
  9. 2019Connecting Vision and Language with Localized Narrativescited by 226
  10. 2019Hyperbolic Image Embeddingscited by 221
  11. 2018Riemannian Adaptive Optimization Methodscited by 194
  12. 2018Zero-Shot Object Detectioncited by 189
  13. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  14. 2018Hyperbolic Entailment Cones for Learning Hierarchical Embeddingscited by 162
  15. 2018Poincar'e GloVe: Hyperbolic Word Embeddingscited by 156
  16. 2021Hyperbolic Deep Neural Networks: A Surveycited by 139
  17. 2023Hyperbolic Image-Text Representationscited by 136
  18. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  19. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  20. 2015Order-Embeddings of Images and Languagecited by 113
  21. 2022Hyperbolic Image Segmentationcited by 102
  22. 2018Embedding Text in Hyperbolic Spacescited by 76
  23. 2023Hyperbolic Deep Learning in Computer Vision: A Surveycited by 73
  24. 2021Unsupervised Object-Level Representation Learning from Scene Imagescited by 72
  25. 2020Self-Supervised Visual Representation Learning from Hierarchical Groupingcited by 63
  26. 2022Hyperbolic Contrastive Learning for Visual Representations beyond Objectscited by 61
  27. 2017Neural Embeddings of Graphs in Hyperbolic Spacecited by 58
  28. 2017Fine-graind Image Classification via Combining Vision and Languagecited by 51
  29. 2019Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddingscited by 49
  30. 2021HoroPCA: Hyperbolic Dimensionality Reduction via Horospherical Projectionscited by 47
  31. 2013Evaluation Measures for Hierarchical Classification: a unified view and novel approachescited by 46
  32. 2023HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representationscited by 44
  33. 2021CO-SNE: Dimensionality Reduction and Visualization for Hyperbolic Datacited by 28
  34. 2017Hierarchical Embeddings for Hypernymy Detection and Directionalitycited by 19
  35. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  36. 2017Specialising Word Vectors for Lexical Entailmentcited by 18
  37. 2023Hyperbolic Active Learning for Semantic Segmentation under Domain Shiftcited by 18
  38. 2023Contracting Skeletal Kinematics for Human-Related Video Anomaly Detectioncited by 14

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 28; 2026: 45.

74 papers.

Sort:

By area:

  1. 2025HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Expertscited by 26
  2. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 21
  3. 2025Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometriescited by 18
  4. 2025Hyperbolic Safety-Aware Vision-Language Modelscited by 17
  5. 2025HLFormer: Enhancing Partially Relevant Video Retrieval with Hyperbolic Learningcited by 17
  6. 2025Hyperbolic Dataset Distillationcited by 15
  7. 2024Learning Visual Hierarchies in Hyperbolic Space for Image Retrievalcited by 8
  8. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  9. 2025PHyCLIP: ℓ1-Product of Hyperbolic Factors Unifies Hierarchy and Compositionality in Vision-Language Representation Learningcited by 7
  10. 2026GeoWorld: Geometric World Modelscited by 6
  11. 2025HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Modelscited by 5
  12. 2026Hyperbolic Busemann Neural Networkscited by 5
38 more
  1. 2026HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignmentcited by 5
  2. 2025Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learningcited by 4
  3. 2026Path-Decoupled Hyperbolic Flow Matching for Few-Shot Adaptationcited by 4
  4. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 4
  5. 2025ProbRes: Probabilistic Jump Diffusion for Open-World Egocentric Activity Recognitioncited by 3
  6. 2025Global and Local Entailment Learning for Natural World Imagerycited by 3
  7. 2025Modality Alignment across Trees on Heterogeneous Hyperbolic Manifoldscited by 3
  8. 2025NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimizationcited by 3
  9. 2025Understanding and Improving Hyperbolic Deep Reinforcement Learningcited by 3
  10. 2026Intrinsic Lorentz Neural Networkcited by 3
  11. 2025HyperPath: Knowledge-Guided Hyperbolic Semantic Hierarchy Modeling for WSI Analysiscited by 2
  12. 2025Learning Protein-Ligand Binding in Hyperbolic Spacecited by 2
  13. 2025Angular Gradient Sign Method: Uncovering Vulnerabilities in Hyperbolic Networkscited by 2
  14. 2026Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Modelscited by 2
  15. 2026Hyperbolic Multimodal Continual Learningcited by 2
  16. 2025DELST: Dual Entailment Learning for Hyperbolic Image-Gene Pretraining in Spatial Transcriptomicscited by 1
  17. 2025Hyperbolic Cycle Alignment for Infrared-Visible Image Fusioncited by 1
  18. 2025H2em: Learning Hierarchical Hyperbolic Embeddings for Compositional Zero-Shot Learningcited by 1
  19. 2026HyperAlign: Hyperbolic Entailment Cones for Adaptive Text-to-Image Alignment Assessmentcited by 1
  20. 2026HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generationcited by 1
  21. 2026Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Expertscited by 1
  22. 2026ARGENT: Adaptive Hierarchical Image-Text Representationscited by 1
  23. 2026Harnessing Hyperbolic Geometry for Harmful Prompt Detection and Sanitizationcited by 1
  24. 2026Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentationcited by 1
  25. 2026Attend to Anything: Foundation Model for Unified Human Attention Modelingcited by 1
  26. 2026HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoningcited by 1
  27. 2026Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localizationcited by 1
  28. 2026HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Spacecited by 1
  29. 2026The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Modelscited by 1
  30. 2026hyperbolix: Hyperbolic Deep Learning in JAXcited by 1
  31. 2025Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learningcited by 0
  32. 2025Hyperbolic Multimodal Representation Learning for Biological Taxonomiescited by 0
  33. 2025Talk in Pieces, See in Whole: Disentangled and Hierarchical Representation Learning in Language-based Object Detectioncited by 0
  34. 2025HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modelingcited by 0
  35. 2025More Than A Shortcut: A Hyperbolic Approach To Early-Exit Networkscited by 0
  36. 2025HyperST: Hierarchical Hyperbolic Learning for Spatial Transcriptomics Predictioncited by 0
  37. 2025Training-Free Dual Hyperbolic Adapters for Better Cross-Modal Reasoningcited by 0
  38. 2026Learning Consistent Taxonomic Classification through Hierarchical Reasoningcited by 0

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
  2. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Models
  3. 2026HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space
  4. 2025Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
  5. 2024Prompting Large Vision-Language Models for Compositional Reasoning
  6. 2026Inference-Time Structural Reasoning for Compositional Vision-Language Understanding
  7. 2024The ART of Composition: Attention-Regularized Training for Compositional Visual Grounding
  8. 2024Natural Language Inference Improves Compositionality in Vision-Language Models