mlsanityData to 14 Oct 2026

Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation

arXiv 2303.12112 ↗First posted 21 Mar 2023cs.CV cs.AI cs.CLcited by 78

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The CLIP model has been recently proven to be very effective for a variety of cross-modal tasks, including the evaluation of captions generated from vision-and-language architectures. In this paper, we propose a new recipe for a contrastive-based evaluation metric for image captioning, namely Positive-Augmented Contrastive learning Score (PAC-S), that in a novel way unifies the learning of a contrastive visual-semantic space with the addition of generated images and text on curated data. Experiments spanning several datasets demonstrate that our new metric achieves the highest correlation with human judgments on both images and videos, outperforming existing reference-based metrics like CIDEr and SPICE and reference-free metrics like CLIP-Score. Finally, we test the system-level correlation of the proposed metric when considering popular image captioning approaches, and assess the impact of employing different cross-modal features. Our source code and trained models are publicly available at: https://github.com/aimagelab/pacscore.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 53 of its 75 references that are in the map. First posted on arXiv 21 Mar 2023.

Builds on

53 of 75 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  11. 2021Robust fine-tuning of zero-shot modelscited by 797
  12. 2018Object Hallucination in Image Captioningcited by 669
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  14. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  15. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  16. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  17. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  18. 2019Attention on Attention for Image Captioningcited by 289
  19. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  20. 2020X-Linear Attention Networks for Image Captioningcited by 198
  21. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  22. 2019Image Captioning: Transforming Objects into Wordscited by 171
  23. 2017Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Imagescited by 101
  24. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  25. 2018Grounded Video Descriptioncited by 78
  26. 2016Boosting Image Captioning with Attributescited by 76
  27. 2018Show, Control and Tell: A Framework for Generating Controllable and Grounded Captionscited by 65
  28. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  29. 2022Disentangling visual and written concepts in CLIPcited by 58
  30. 2018Learning to Evaluate Image Captioningcited by 53
  31. 2019Describing like humans: on diversity in image captioningcited by 46
  32. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  33. 2021UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learningcited by 37
  34. 2021EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matchingcited by 36
  35. 2015From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledgecited by 35
  36. 2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
  37. 2020Towards Unique and Informative Captioning of Imagescited by 25
  38. 2022CaMEL: Mean Teacher Learning for Image Captioningcited by 14

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 25; 2025: 31; 2026: 17.

78 papers.

Sort:

By area:

  1. 2024Benchmarking and Improving Detail Image Captioncited by 81
  2. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  3. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  4. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 45
  5. 2023Parents and Children: Distinguishing Multimodal DeepFakes from Natural Imagescited by 29
  6. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learningcited by 25
  7. 2024G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4ocited by 23
  8. 2023Mitigating Open-Vocabulary Caption Hallucinationscited by 19
  9. 2023With a Little Help from your own Past: Prototypical Memory Networks for Image Captioningcited by 18
  10. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 18
  11. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  12. 2025SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learningcited by 16
38 more
  1. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  2. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  3. 2023An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metricscited by 11
  4. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  5. 2025Pretrained Image-Text Models are Secretly Video Captionerscited by 11
  6. 2024I0T: Embedding Standardization Method Towards Zero Modality Gapcited by 8
  7. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8
  8. 2023Synaptic Plasticity Models and Bio-Inspired Unsupervised Deep Learning: A Surveycited by 7
  9. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extractioncited by 7
  10. 2024Towards Retrieval-Augmented Architectures for Image Captioningcited by 7
  11. 2024Efficient and Versatile Robust Fine-Tuning of Zero-shot Modelscited by 7
  12. 2025EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanationscited by 7
  13. 2024ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Modelscited by 6
  14. 2025Panoptic Captioning: An Equivalence Bridge for Image and Textcited by 6
  15. 2024SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by Visual-Textual Contrastive Learningcited by 5
  16. 2024Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimizationcited by 5
  17. 2024Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologiescited by 5
  18. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captionscited by 5
  19. 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
  20. 2025ZINA: Multimodal Fine-grained Hallucination Detection and Editingcited by 5
  21. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5
  22. 2025Digital Twin Buildings: 3D Modeling, GIS Integration, and Visual Descriptions Using Gaussian Splatting, ChatGPT/Deepseek, and Google Maps Platformcited by 4
  23. 2025VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysiscited by 4
  24. 2025MRG-R1: Reinforcement Learning for Clinically Aligned Medical Report Generationcited by 4
  25. 2025Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptionscited by 3
  26. 2024BID: Boundary-Interior Decoding for Unsupervised Temporal Action Localization Pre-Trainincited by 2
  27. 2024Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Modelscited by 2
  28. 2024Towards Unified Benchmark and Models for Multi-Modal Perceptual Metricscited by 2
  29. 2025SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluationcited by 2
  30. 2025FinCap: Topic-Aligned Captions for Short-Form Financial YouTube Videoscited by 2
  31. 2025One Patch to Caption Them All: A Unified Zero-Shot Captioning Frameworkcited by 2
  32. 2025DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioningcited by 2
  33. 2025LLM-Free Image Captioning Evaluation in Reference-Flexible Settingscited by 2
  34. 2026Do Image-Text Metrics Respect Semantic Invariances?cited by 2
  35. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1
  36. 2024Enhancing Contrastive Learning Inspired by the Philosophy of "The Blind Men and the Elephant"cited by 1
  37. 2025Can Text-to-Video Generation help Video-Language Alignment?cited by 1
  38. 2025Metropolis-Hastings Captioning Game: Knowledge Fusion of Vision Language Models via Decentralized Bayesian Inferencecited by 1

50 most cited of 78 All 78 (JSON)