mlsanityData to 7 Oct 2026

Discriminability objective for training descriptive captions

arXiv 1803.04376 ↗First posted 12 Mar 2018cs.CVcited by 67

Not on the map. Why?

Abstract (arXiv)

One property that remains lacking in image captions generated by contemporary methods is discriminability: being able to tell two images apart given the caption for one of them. We propose a way to improve this aspect of caption generation. By incorporating into the captioning training objective a loss component directly related to ability (by a machine) to disambiguate image/caption matches, we obtain systems that produce much more discriminative caption, according to human evaluation. Remarkably, our approach leads to improvement in other aspects of generated captions, reflected by a battery of standard scores such as BLEU, SPICE etc. Our approach is modular and can be applied to a variety of model/loss combinations commonly proposed for image captioning.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 19; 2023: 13; 2024: 10; 2025: 4; 2026: 2.

67 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  4. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  5. 2021Causal Attention for Vision-Language Taskscited by 111
  6. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  7. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  8. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  9. 2021Human-like Controllable Image Captioning with Verb-specific Semantic Rolescited by 41
  10. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  11. 2021UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learningcited by 37
  12. 2022Measuring Representational Harms in Image Captioningcited by 36
38 more
  1. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  2. 2021ITA: Image-Text Alignments for Multi-Modal Named Entity Recognitioncited by 24
  3. 2025AVD2: Accident Video Diffusion for Accident Video Descriptioncited by 20
  4. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  5. 2021Learning Structural Representations for Recipe Generation and Food Retrievalcited by 17
  6. 2022Adaptive Fine-Grained Predicates Learning for Scene Graph Generationcited by 14
  7. 2023UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Trainingcited by 14
  8. 2023Cross-Domain Image Captioning with Discriminative Finetuningcited by 14
  9. 2021Group-based Distinctive Image Captioning with Memory Attentioncited by 13
  10. 2024DistinctAD: Distinctive Audio Description Generation in Contextscited by 11
  11. 2022Dual-branch Hybrid Learning Network for Unbiased Scene Graph Generationcited by 10
  12. 2023Effective End-to-End Vision Language Pretraining with Semantic Visual Losscited by 10
  13. 2023Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Modelcited by 10
  14. 2024A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenescited by 10
  15. 2021Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Modelscited by 9
  16. 2022Rethinking the Reference-based Distinctive Image Captioningcited by 8
  17. 2023DisCLIP: Open-Vocabulary Referring Expression Generationcited by 8
  18. 2023LLM4VG: Large Language Models Evaluation for Video Groundingcited by 8
  19. 2021Automatic Generation of Descriptive Titles for Video Clips Using Deep Learningcited by 7
  20. 2022On Distinctive Image Captioning via Comparing and Reweightingcited by 7
  21. 2022Distinctive Image Captioning via CLIP Guided Group Optimizationcited by 6
  22. 2022Text-Guided Mask-free Local Image Retouchingcited by 6
  23. 2023Multimodal Data Augmentation for Image Captioning using Diffusion Modelscited by 6
  24. 2021Counterfactual Maximum Likelihood Estimation for Training Deep Networkscited by 5
  25. 2022CapEnrich: Enriching Caption Semantics for Web Images via Cross-modal Pre-trained Knowledgecited by 5
  26. 2022Look, Read and Ask: Learning to Ask Questions by Reading Text in Imagescited by 5
  27. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioningcited by 5
  28. 2021Cross-Modal Graph with Meta Concepts for Video Captioningcited by 4
  29. 2022Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learningcited by 4
  30. 2023IMAGINATOR: Pre-Trained Image+Text Joint Embeddings using Word-Level Grounding of Imagescited by 4
  31. 2023Pragmatic Inference with a CLIP Listener for Contrastive Captioningcited by 4
  32. 2023Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimationcited by 4
  33. 2023Improving Reference-based Distinctive Image Captioning with Contrastive Rewardscited by 4
  34. 2021Multi-modal Ensemble Models for Predicting Video Memorabilitycited by 3
  35. 2022Towards Automated Error Analysis: Learning to Characterize Errorscited by 3
  36. 2022Cross-Modal Similarity-Based Curriculum Learning for Image Captioningcited by 3
  37. 2024Image Captioning via Dynamic Path Customizationcited by 3
  38. 2022Predicting Word Learning in Children from the Performance of Computer Vision Systemscited by 2

50 most cited of 67 All 67 (JSON)