mlsanityData to 7 Oct 2026

Alpha-CLIP: A CLIP Model Focusing on Wherever You Want

arXiv 2312.03818 ↗First posted 6 Dec 2023cs.CV cs.AI cs.CLcited by 150

Not on the map. Why?

Cited by 50 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) plays an essential role in extracting valuable content information from images across diverse tasks. It aligns textual and visual modalities to comprehend the entire image, including all the details, even those irrelevant to specific tasks. However, for a finer understanding and controlled editing of images, it becomes crucial to focus on specific regions of interest, which can be indicated as points, masks, or boxes by humans or perception models. To fulfill the requirements, we introduce Alpha-CLIP, an enhanced version of CLIP with an auxiliary alpha channel to suggest attentive regions and fine-tuned with constructed millions of RGBA region-text pairs. Alpha-CLIP not only preserves the visual recognition ability of CLIP but also enables precise control over the emphasis of image contents. It demonstrates effectiveness in various tasks, including but not limited to open-world recognition, multimodal large language models, and conditional 2D / 3D generation. It has a strong potential to serve as a versatile tool for image-related tasks.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Builds on 63 of its 76 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

63 of 76 references.

References · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2017Mask R-CNNcited by 7,259
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  13. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  14. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  15. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  16. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  17. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  18. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  19. 2021Extract Free Dense Labels from CLIPcited by 580
  20. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  21. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  22. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  23. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  24. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  25. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  26. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  27. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  28. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  29. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  30. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  31. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  32. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  33. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  34. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  35. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  36. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  37. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  38. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 50; 2025: 68; 2026: 28.

150 papers.

Sort:

By area:

  1. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  2. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  3. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  4. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  5. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 85
  6. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  7. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  9. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 63
  10. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  11. 2025SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Imagescited by 41
  12. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
38 more
  1. 2024VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Modelscited by 40
  2. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  3. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  4. 2024FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instructioncited by 35
  5. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  6. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  7. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  8. 2023Tokenize Anything via Promptingcited by 31
  9. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMscited by 29
  10. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 26
  11. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 26
  12. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  13. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  14. 2025SmartEraser: Remove Anything from Images using Masked-Region Guidancecited by 24
  15. 2024Advancing Textual Prompt Learning with Anchored Attributescited by 20
  16. 2024CLIPAway: Harmonizing Focused Embeddings for Removing Objects via Diffusion Modelscited by 19
  17. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  18. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  19. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 17
  20. 2024Improving Medical Multi-modal Contrastive Learning with Expert Annotationscited by 16
  21. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  22. 2025Top-Down Semantic Refinement for Image Captioningcited by 16
  23. 2024VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Textcited by 15
  24. 2024Effective SAM Combination for Open-Vocabulary Semantic Segmentationcited by 15
  25. 2025ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Modelscited by 15
  26. 2024DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Modelscited by 14
  27. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  28. 2024ControlCap: Controllable Region-level Captioningcited by 13
  29. 2024CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenescited by 13
  30. 2024Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspectivecited by 13
  31. 2023Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Casescited by 12
  32. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  33. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  34. 2024Open-Vocabulary Online Semantic Mapping for SLAMcited by 11
  35. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 11
  36. 2023Long-Tailed 3D Detection via Multi-Modal Fusioncited by 10
  37. 2025Are We Done with Object-Centric Learning?cited by 10
  38. 2025Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modelingcited by 10

50 most cited of 150 All 150 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
  2. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  3. 2023Raising the Bar of AI-generated Image Detection with CLIP
  4. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study
  5. 2025CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation
  6. 2025un2CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP
  7. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  8. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models