# Image Embedding Sampling Method for Diverse Captioning

arXiv: https://arxiv.org/abs/2502.10118 · HTML: https://mlsanity.com/paper/2502.10118/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [Honeybee: Locality-enhanced Projector for Multimodal LLM](https://mlsanity.com/paper/2312.06742/) (2023)
- [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](https://mlsanity.com/paper/2311.05437/) (2023)
- [Learning non-maximum suppression](https://mlsanity.com/paper/1705.02950/) (2017)
- [Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training](https://mlsanity.com/paper/2210.08773/) (2022)
- [Zero-shot Referring Image Segmentation with Global-Local Context Features](https://mlsanity.com/paper/2303.17811/) (2023)
- [Hierarchy Parsing for Image Captioning](https://mlsanity.com/paper/1909.03918/) (2019)
- [Injecting Semantic Concepts into End-to-End Image Captioning](https://mlsanity.com/paper/2112.05230/) (2021)
- [Step-Wise Hierarchical Alignment Network for Image-Text Matching](https://mlsanity.com/paper/2106.06509/) (2021)
- [Towards Accurate Text-based Image Captioning with Content Diversity Exploration](https://mlsanity.com/paper/2105.03236/) (2021)
- [Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning](https://mlsanity.com/paper/1908.08529/) (2019)
- [Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusion](https://mlsanity.com/paper/2306.11593/) (2023)
- [Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment](https://mlsanity.com/paper/2208.13628/) (2022)
- [Diffusion Models Through a Global Lens: Are They Culturally Inclusive?](https://mlsanity.com/paper/2502.08914/) (2025)
- [Learning Distinct and Representative Styles for Image Captioning](https://mlsanity.com/paper/2209.08231/) (2022)

## Abstract (arXiv)

Image Captioning for state-of-the-art VLMs has significantly improved over time; however, this comes at the cost of increased computational complexity, making them less accessible for resource-constrained applications such as mobile devices and assistive technologies. Alternatively, comparably smaller VLMs prioritize high-level scene descriptions, overlooking finer details that contribute to a richer understanding of an image. In this paper, we introduce a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM, BLIP, as the backbone. Our approach leverages structured segmentation to produce hierarchical representations that capture both global and localized semantics. Without requiring additional model training, we demonstrate that our method allows smaller VLMs to achieve performance comparable to larger models in terms of image-caption alignment, semantic integrity, and diversity. We evaluate our framework on MSCOCO, Flickr30k, and Nocaps test datasets, achieving a Div-2 score of 0.735, 0.750, and 0.748 for each dataset, respectively, while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions.