# PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning

arXiv: https://arxiv.org/abs/2511.04601 · HTML: https://mlsanity.com/paper/2511.04601/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 6 Nov 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [Improving CLIP Training with Language Rewrites](https://mlsanity.com/paper/2305.20088/) (2023)
- [Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2302.03084/) (2023)
- [Alpha-CLIP: A CLIP Model Focusing on Wherever You Want](https://mlsanity.com/paper/2312.03818/) (2023)
- [VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks](https://mlsanity.com/paper/2406.08394/) (2024)
- [DOCCI: Descriptions of Connected and Contrasting Images](https://mlsanity.com/paper/2404.19753/) (2024)
- [ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension](https://mlsanity.com/paper/2204.05991/) (2022)
- [ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference](https://mlsanity.com/paper/2407.12442/) (2024)
- [Large-scale Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2106.03149/) (2021)
- [DreamLIP: Language-Image Pre-training with Long Captions](https://mlsanity.com/paper/2403.17007/) (2024)
- [OvarNet: Towards Open-vocabulary Object Attribute Recognition](https://mlsanity.com/paper/2301.09506/) (2023)
- [Contrastive Localized Language-Image Pre-Training](https://mlsanity.com/paper/2410.02746/) (2024)

## Built on by (most cited first, 5 of 5)

- [OmniReasoner: Thinking with Long Audio-Video via Native Tool Use](https://mlsanity.com/paper/2607.19339/) (2026)
- [Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding](https://mlsanity.com/paper/2602.02977/) (2026)
- [Membership Inference for Contrastive Pre-training Models with Text-only PII Queries](https://mlsanity.com/paper/2603.14222/) (2026)
- [RADA: Region-Aware Dual-encoder Auxiliary learning for Barely-supervised Medical Image Segmentation](https://mlsanity.com/paper/2604.11164/) (2026)
- [Achieving Text-based Person Retrieval with Any Granularity](https://mlsanity.com/paper/2607.21057/) (2026)

## Abstract (arXiv)

While the Contrastive Language-Image Pretraining(CLIP) model has achieved remarkable success in a variety of downstream vison language understanding tasks, enhancing its capability for fine-grained image-text alignment remains an active research focus. To this end, most existing works adopt the strategy of explicitly increasing the granularity of visual information processing, e.g., incorporating visual prompts to guide the model focus on specific local regions within the image. Meanwhile, researches on Multimodal Large Language Models(MLLMs) have demonstrated that training with long and detailed textual descriptions can effectively improve the model's fine-grained vision-language alignment. However, the inherent token length limitation of CLIP's text encoder fundamentally limits CLIP to process more granular textual information embedded in long text sequences. To synergistically leverage the advantages of enhancing both visual and textual content processing granularity, we propose PixCLIP, a novel framework designed to concurrently accommodate visual prompt inputs and process lengthy textual descriptions. Specifically, we first establish an automated annotation pipeline capable of generating pixel-level localized, long-form textual descriptions for images. Utilizing this pipeline, we construct LongGRIT, a high-quality dataset comprising nearly 1.5 million samples. Secondly, we replace CLIP's original text encoder with the LLM and propose a three-branch pixel-text alignment learning framework, facilitating fine-grained alignment between image regions and corresponding textual descriptions at arbitrary granularity. Experiments demonstrate that PixCLIP showcases breakthroughs in pixel-level interaction and handling long-form texts, achieving state-of-the-art performance.