mlsanityData to 7 Oct 2026

Unifying Vision-Language Latents for Zero-label Image Caption Enhancement

arXiv 2510.12931 ↗First posted 14 Oct 2025cs.CV cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs) achieve remarkable performance through large-scale image-text pretraining. However, their reliance on labeled image datasets limits scalability and leaves vast amounts of unlabeled image data underutilized. To address this, we propose Unified Vision-Language Alignment for Zero-Label Enhancement (ViZer), an enhancement training framework that enables zero-label learning in image captioning, providing a practical starting point for broader zero-label adaptation in vision-language tasks. Unlike prior approaches that rely on human or synthetically annotated datasets, ViZer actively aligns vision and language representation features during training, enabling existing VLMs to generate improved captions without requiring text labels or full retraining. We demonstrate ViZer's advantage in qualitative evaluation, as automated caption metrics such as CIDEr and BERTScore often penalize details that are absent in reference captions. Applying ViZer on SmolVLM-Base and Qwen2-VL, we observe consistent qualitative improvements, producing captions that are more grounded and descriptive than their baseline.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 44 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

34 of 44 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
22 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  7. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  8. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  12. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  13. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  14. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  15. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  16. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  17. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  18. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 53
  19. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  20. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  21. 2025Evaluating Image Caption via Cycle-consistent Text-to-Image Generationcited by 4
  22. 2025L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Trainingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026UniCompress: Token Compression for Unified Vision-Language Understanding and Generationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
  2. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
  3. 2023UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding
  4. 2025Synthetic Captions for Open-Vocabulary Zero-Shot Segmentation
  5. 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models
  6. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
  7. 2022Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment
  8. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretraining