# Style-Aware Contrastive Learning for Multi-Style Image Captioning

arXiv: https://arxiv.org/abs/2301.11367 · HTML: https://mlsanity.com/paper/2301.11367/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [Optimizing Dense Retrieval Model Training with Hard Negatives](https://mlsanity.com/paper/2104.08051/) (2021)
- [Engaging Image Captioning Via Personality](https://mlsanity.com/paper/1810.10665/) (2018)
- [SentiCap: Generating Image Descriptions with Sentiments](https://mlsanity.com/paper/1510.01431/) (2015)
- [ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification](https://mlsanity.com/paper/2203.02225/) (2022)
- [SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text](https://mlsanity.com/paper/1805.07030/) (2018)
- ["Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention](https://mlsanity.com/paper/1807.03871/) (2018)
- [Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning](https://mlsanity.com/paper/2108.11912/) (2021)
- [Structural and Functional Decomposition for Personality Image Captioning in a Communication Game](https://mlsanity.com/paper/2011.08543/) (2020)
- [3M: Multi-style image caption generation using Multi-modality features under Multi-UPDOWN model](https://mlsanity.com/paper/2103.11186/) (2021)

## Built on by (most cited first, 21 of 21)

- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook](https://mlsanity.com/paper/2503.18016/) (2025)
- [MemoryMamba: Memory-Augmented State Space Model for Defect Recognition](https://mlsanity.com/paper/2405.03673/) (2024)
- [Understanding Retrieval Robustness for Retrieval-Augmented Image Captioning](https://mlsanity.com/paper/2406.02265/) (2024)
- [Exploiting Diffusion Prior for Out-of-Distribution Detection](https://mlsanity.com/paper/2406.11105/) (2024)
- [An LLM-LVLM Driven Agent for Iterative and Fine-Grained Image Editing](https://mlsanity.com/paper/2508.17435/) (2025)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Large Language Models for Zero-Shot Multicultural Name Recognition](https://mlsanity.com/paper/2507.04149/) (2025)
- [LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation](https://mlsanity.com/paper/2508.04732/) (2025)
- [XDR-LVLM: An Explainable Vision-Language Large Model for Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2508.15168/) (2025)
- [PoultryTalk: A Multi-modal Retrieval-Augmented Generation (RAG) System for Intelligent Poultry Management and Decision Support](https://mlsanity.com/paper/2512.08995/) (2025)
- [Exploring Large Vision-Language Models for Robust and Efficient Industrial Anomaly Detection](https://mlsanity.com/paper/2412.00890/) (2024)
- [Reasoning-Aware Query-Focused Summarization over Multi-Table Data](https://mlsanity.com/paper/2412.08970/) (2024)
- [Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives](https://mlsanity.com/paper/2412.10720/) (2024)
- [Optimizing Vision-Language Interactions Through Decoder-Only Models](https://mlsanity.com/paper/2412.10758/) (2024)
- [Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes](https://mlsanity.com/paper/2412.11396/) (2024)
- [Hierarchical Vision-Language Alignment for Text-to-Image Generation via Diffusion Models](https://mlsanity.com/paper/2501.00917/) (2025)
- [Weak Supervision Dynamic KL-Weighted Diffusion Models Guided by Large Language Models](https://mlsanity.com/paper/2502.00826/) (2025)
- [A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography](https://mlsanity.com/paper/2502.05926/) (2025)
- [Revolutionizing Radiology Workflow with Factual and Efficient CXR Report Generation](https://mlsanity.com/paper/2506.01118/) (2025)
- [When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPO](https://mlsanity.com/paper/2610.06861/) (2026)

## Abstract (arXiv)

Existing multi-style image captioning methods show promising results in generating a caption with accurate visual content and desired linguistic style. However, existing methods overlook the relationship between linguistic style and visual content. To overcome this drawback, we propose style-aware contrastive learning for multi-style image captioning. First, we present a style-aware visual encoder with contrastive learning to mine potential visual content relevant to style. Moreover, we propose a style-aware triplet contrast objective to distinguish whether the image, style and caption matched. To provide positive and negative samples for contrastive learning, we present three retrieval schemes: object-based retrieval, RoI-based retrieval and triplet-based retrieval, and design a dynamic trade-off function to calculate retrieval scores. Experimental results demonstrate that our approach achieves state-of-the-art performance. In addition, we conduct an extensive analysis to verify the effectiveness of our method.