mlsanityData to 14 Oct 2026

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

arXiv 2303.02995 ↗First posted 6 Mar 2023cs.CV cs.CL cs.LGcited by 44

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other vision-language models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning. To this end, we equip both the visual and language branches in CLIP with hierarchy-aware attentions, namely Hierarchy-aware CLIP (HiCLIP), to progressively discover semantic hierarchies layer-by-layer from both images and texts in an unsupervised manner. As a result, such hierarchical aggregation significantly improves the cross-modal alignment. To demonstrate the advantages of HiCLIP, we conduct qualitative analysis on its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 73 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

46 of 73 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2013Describing Textures in the Wildcited by 2,125
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
32 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  5. 2020Object-Centric Learning with Slot Attentioncited by 876
  6. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  8. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  9. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  10. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  11. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  12. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  13. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  14. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  15. 2018Bilinear Attention Networkscited by 280
  16. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  17. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  18. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  19. 2021DetCo: Unsupervised Contrastive Learning for Object Detectioncited by 205
  20. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  21. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  22. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  23. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  24. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  25. 2019Graphical Contrastive Losses for Scene Graph Parsingcited by 100
  26. 2020Feature Pyramid Transformercited by 96
  27. 2019Compound Probabilistic Context-Free Grammars for Grammar Inductioncited by 76
  28. 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
  29. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  30. 2019Visually Grounded Neural Syntax Acquisitioncited by 48
  31. 2022Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervisioncited by 36
  32. 2020Visually Grounded Compound PCFGscited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 19; 2025: 6; 2026: 6.

44 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  3. 2023Improving Tuning-Free Real Image Editing with Proximal Guidancecited by 87
  4. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  5. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  6. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  7. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  8. 2023VIP5: Towards Multimodal Foundation Models for Recommendationcited by 46
  9. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  10. 2023Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokenscited by 35
  11. 2023Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysiscited by 34
  12. 2023Image Clustering Conditioned on Text Criteriacited by 26
32 more
  1. 2024RWKV-CLIP: A Robust Vision-Language Representation Learnercited by 23
  2. 2024CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discriminationcited by 21
  3. 2024Law of Vision Representation in MLLMscited by 18
  4. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  5. 2023Free-Editor: Zero-shot Text-driven 3D Scene Editingcited by 14
  6. 2023Cross-Modal Concept Learning and Inference for Vision-Language Modelscited by 12
  7. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  8. 2023Misalign, Contrast then Distill: Rethinking Misalignments in Language-Image Pretrainingcited by 10
  9. 2024Toward Interactive Regional Understanding in Vision-Large Language Modelscited by 9
  10. 2024RankCLIP: Ranking-Consistent Language-Image Pretrainingcited by 9
  11. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  12. 2023CAPIVARA: Cost-Efficient Approach for Improving Multilingual CLIP Performance on Low-Resource Languagescited by 7
  13. 2024FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language modelscited by 7
  14. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  15. 2023Weakly-Supervised Learning of Visual Relations in Multimodal Pretrainingcited by 6
  16. 2023BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoningcited by 6
  17. 2024MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilizationcited by 6
  18. 2024Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignmentcited by 6
  19. 2024Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editingcited by 6
  20. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 4
  21. 2025Towards Self-Refinement of Vision-Language Models with Triangular Consistencycited by 3
  22. 2024The Devil is in the Few Shots: Iterative Visual Knowledge Completion for Few-shot Learningcited by 1
  23. 2025LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matchingcited by 1
  24. 2025MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusioncited by 1
  25. 2026Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understandingcited by 1
  26. 2024GlocalCLIP: Object-agnostic Global-Local Prompt Learning for Zero-shot Anomaly Detectioncited by 0
  27. 2024Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selectioncited by 0
  28. 2025Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervisioncited by 0
  29. 2026Learning Consistent Taxonomic Classification through Hierarchical Reasoningcited by 0
  30. 2026HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understandingcited by 0
  31. 2026Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Modelscited by 0
  32. 2026Is the Geometry Doing the Work? An Operating-Point Audit of Hierarchy in Hyperbolic Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining
  2. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  3. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  4. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  5. 2025DiffCLIP: Differential Attention Meets CLIP
  6. 2026HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space
  7. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
  8. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations