mlsanityData to 7 Oct 2026

Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights

arXiv 2407.11449 ↗First posted 16 Jul 2024cs.CV cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Contextualized Image Captioning (CIC) evolves traditional image captioning into a more complex domain, necessitating the ability for multimodal reasoning. It aims to generate image captions given specific contextual information. This paper further introduces a novel domain of Controllable Contextualized Image Captioning (Ctrl-CIC). Unlike CIC, which solely relies on broad context, Ctrl-CIC accentuates a user-defined highlight, compelling the model to tailor captions that resonate with the highlighted aspects of the context. We present two approaches, Prompting-based Controller (P-Ctrl) and Recalibration-based Controller (R-Ctrl), to generate focused captions. P-Ctrl conditions the model generation on highlight by prepending captions with highlight-driven prefixes, whereas R-Ctrl tunes the model to selectively recalibrate the encoder embeddings for highlighted tokens. Additionally, we design a GPT-4V empowered evaluator to assess the quality of the controlled captions alongside standard assessment methods. Extensive experimental results demonstrate the efficient and effective controllability of our method, charting a new direction in achieving user-adaptive image captioning. Code is available at https://github.com/ShunqiM/Ctrl-CIC .

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 52 of its 69 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

52 of 69 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
38 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  10. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  11. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  12. 2020X-Linear Attention Networks for Image Captioningcited by 198
  13. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  14. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  15. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  16. 2020A Distributional Approach to Controlled Text Generationcited by 117
  17. 2018Data-to-Text Generation with Content Selection and Planningcited by 106
  18. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  19. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  20. 2021BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Datacited by 75
  21. 2019Good News, Everyone! Context driven entity-aware captioning for news imagescited by 66
  22. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  23. 2020Towards Persona-Based Empathetic Conversational Modelscited by 53
  24. 2020Transform and Tell: Entity-Aware News Image Captioningcited by 52
  25. 2021Structural Adapters in Pretrained Language Models for AMR-to-text Generationcited by 48
  26. 2020Generate, Delete and Rewrite: A Three-Stage Framework for Improving Persona Consistency of Dialogue Generationcited by 41
  27. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  28. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
  29. 2021Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graphcited by 31
  30. 2021Parallel Refinements for Lexically Constrained Text Generation with BARTcited by 24
  31. 2019Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioningcited by 21
  32. 2016BreakingNews: Article Annotation by Image and Text Processingcited by 20
  33. 2023A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understandingcited by 18
  34. 2022Controllable Image Captioning via Promptingcited by 15
  35. 2021Journalistic Guidelines Aware News Image Captioningcited by 14
  36. 2023Visually-Aware Context Modeling for News Image Captioningcited by 12
  37. 2022Show, Interpret and Tell: Entity-aware Contextualised Image Captioning in Wikipediacited by 10
  38. 2022Focus! Relevant and Sufficient Context Selection for News Image Captioningcited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5
  2. 2025ImageSet2Text: Describing Sets of Images through Textcited by 0
  3. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contextscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Controllable Image Captioning with Prompt-Conditioned Scene Rewards
  2. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
  3. 2026CIAN: Multi-Stage Framework for Event-Enriched Image Captioning via Retrieval-Augmented Generation
  4. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
  5. 2025RONA: Pragmatically Diverse Image Captioning with Coherence Relations
  6. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
  7. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models
  8. 2025CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning