# HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts

arXiv: https://arxiv.org/abs/2404.17507 · HTML: https://mlsanity.com/paper/2404.17507/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 79 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 30 of 59)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [Visual Entailment: A Novel Task for Fine-Grained Image Understanding](https://mlsanity.com/paper/1901.06706/) (2019)
- [Data Filtering Networks](https://mlsanity.com/paper/2309.17425/) (2023)

## Built on by (most cited first, 17 of 17)

- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning](https://mlsanity.com/paper/2405.19547/) (2024)
- [Active Data Curation Effectively Distills Large-Scale Multimodal Models](https://mlsanity.com/paper/2411.18674/) (2024)
- [Probabilistic Language-Image Pre-Training](https://mlsanity.com/paper/2410.18857/) (2024)
- [Multilingual Diversity Improves Vision-Language Representations](https://mlsanity.com/paper/2405.16915/) (2024)
- [Quality over Quantity: Boosting Data Efficiency Through Ensembled Multimodal Data Curation](https://mlsanity.com/paper/2502.08211/) (2025)
- [Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning](https://mlsanity.com/paper/2505.19614/) (2025)
- [Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining](https://mlsanity.com/paper/2503.08805/) (2025)
- [Hyperbolic Multimodal Continual Learning](https://mlsanity.com/paper/2608.09572/) (2026)
- [LongProLIP: A Probabilistic Vision-Language Model with Long Context Text](https://mlsanity.com/paper/2503.08048/) (2025)
- [ReSpec: Relevance and Specificity Grounded Online Filtering for Learning on Video-Text Data Streams](https://mlsanity.com/paper/2504.14875/) (2025)
- [ARGENT: Adaptive Hierarchical Image-Text Representations](https://mlsanity.com/paper/2603.23311/) (2026)
- [Understanding the Gain from Data Filtering in Multimodal Contrastive Learning](https://mlsanity.com/paper/2512.14230/) (2025)
- [TADS: Task-Aware Data Selection for Multi-Task Multimodal Pre-Training](https://mlsanity.com/paper/2602.05251/) (2026)
- [Curvature-Aware Captioning:Leveraging Geodesic Attention for 3D Scene Understanding](https://mlsanity.com/paper/2605.08808/) (2026)
- [Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation](https://mlsanity.com/paper/2606.29464/) (2026)
- [Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution](https://mlsanity.com/paper/2609.29329/) (2026)

## Abstract (arXiv)

In an era where the volume of data drives the effectiveness of self-supervised learning, the specificity and clarity of data semantics play a crucial role in model training. Addressing this, we introduce HYPerbolic Entailment filtering (HYPE), a novel methodology designed to meticulously extract modality-wise meaningful and well-aligned data from extensive, noisy image-text pair datasets. Our approach leverages hyperbolic embeddings and the concept of entailment cones to evaluate and filter out samples with meaningless or underspecified semantics, focusing on enhancing the specificity of each data sample. HYPE not only demonstrates a significant improvement in filtering efficiency but also sets a new state-of-the-art in the DataComp benchmark when combined with existing filtering techniques. This breakthrough showcases the potential of HYPE to refine the data selection process, thereby contributing to the development of more accurate and efficient self-supervised learning models. Additionally, the image specificity εi can be independently applied to induce an image-only dataset from an image-text or image-only data pool for training image-only self-supervised models and showed superior performance when compared to the dataset induced by CLIP score.