# β-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment

arXiv: https://arxiv.org/abs/2512.12678 · HTML: https://mlsanity.com/paper/2512.12678/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 14 Dec 2025.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2208.12262/) (2022)
- [DOCCI: Descriptions of Connected and Contrasting Images](https://mlsanity.com/paper/2404.19753/) (2024)
- [Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning](https://mlsanity.com/paper/2212.04994/) (2022)
- [A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions](https://mlsanity.com/paper/2312.08578/) (2023)
- [FLAIR: VLM with Fine-grained Language-informed Image Representations](https://mlsanity.com/paper/2412.03561/) (2024)
- [Implicit Temporal Modeling with Learnable Alignment for Video Recognition](https://mlsanity.com/paper/2304.10465/) (2023)
- [Improving fine-grained understanding in image-text pre-training](https://mlsanity.com/paper/2401.09865/) (2024)
- [The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding](https://mlsanity.com/paper/2311.17518/) (2023)
- [FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs](https://mlsanity.com/paper/2504.01916/) (2025)
- [TULIP: Token-length Upgraded CLIP](https://mlsanity.com/paper/2410.10034/) (2024)
- [SmartCLIP: Modular Vision-language Alignment with Identification Guarantees](https://mlsanity.com/paper/2507.22264/) (2025)
- [Is CLIP the main roadblock for fine-grained open-world perception?](https://mlsanity.com/paper/2404.03539/) (2024)
- [Refining CLIP's Spatial Awareness: A Visual-Centric Perspective](https://mlsanity.com/paper/2504.02328/) (2025)

## Built on by (most cited first, 5 of 5)

- [Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning](https://mlsanity.com/paper/2604.05834/) (2026)
- [TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models](https://mlsanity.com/paper/2603.06999/) (2026)
- [Sparse Attention for Dense Open-Vocabulary Prediction in CLIP](https://mlsanity.com/paper/2607.07135/) (2026)
- [Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognition](https://mlsanity.com/paper/2607.19889/) (2026)
- [Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training](https://mlsanity.com/paper/2608.29313/) (2026)

## Abstract (arXiv)

CLIP achieves strong zero-shot image-text retrieval by aligning global vision and text representations, yet it falls behind on fine-grained tasks even when fine-tuned on long, detailed captions. In this work, we propose β-CLIP, a multi-granular text-conditioned contrastive learning framework designed to achieve hierarchical alignment between multiple textual granularities-from full captions to sentences and phrases-and their corresponding visual regions. For each level of granularity, β-CLIP utilizes cross-attention to dynamically pool image patches, producing contextualized visual embeddings. To address the semantic overlap inherent in this hierarchy, we introduce the β-Contextualized Contrastive Alignment Loss (β-CAL). This objective parameterizes the trade-off between strict query-specific matching and relaxed intra-image contextualization, supporting both soft Cross-Entropy and hard Binary Cross-Entropy formulations. We find that each loss interacts differently with hierarchical supervision: CE's softmax sharpens fine-grained discrimination, while BCE's sigmoid favors long-text retrieval while both benefit from hierarchy. Through extensive experiments, we demonstrate that β-CLIP significantly improves dense alignment: achieving 91.8% T2I 92.3% I2T at R@1 on Urban1K and 30.9% on FG-OVD (Hard), setting state-of-the-art among methods trained without hard negatives. β-CLIP establishes a robust, adaptive baseline for dense vision-language correspondence. The code and models are released at https://github.com/fzohra/B-CLIP.