mlsanityData to 14 Oct 2026

Exploring Diverse In-Context Configurations for Image Captioning

arXiv 2305.14800 ↗First posted 24 May 2023cs.CVcited by 63

Not on the map. Why?

Cited by 38 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

After discovering that Language Models (LMs) can be good in-context few-shot learners, numerous strategies have been proposed to optimize in-context sequence configurations. Recently, researchers in Vision-Language (VL) domains also develop their few-shot learners, while they only use the simplest way, ie., randomly sampling, to configure in-context image-text pairs. In order to explore the effects of varying configurations on VL in-context learning, we devised four strategies for image selection and four for caption assignment to configure in-context image-text pairs for image captioning. Here Image Captioning is used as the case study since it can be seen as the visually-conditioned LM. Our comprehensive experiments yield two counter-intuitive but valuable insights, highlighting the distinct characteristics of VL in-context learning due to multi-modal synergy, as compared to the NLP case. Furthermore, in our exploration of optimal combination strategies, we observed an average performance enhancement of 20.9 of CIDEr scores compared to the baseline. The code is given in https://github.com/yongliang-wu/ExploreCfg.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

51 of 60 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  13. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  14. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  15. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  16. 2019How Can We Know What Language Models Know?cited by 959
  17. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  18. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  19. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  20. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  21. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  22. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  23. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  24. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  25. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  26. 2023Augmented Language Models: a Surveycited by 403
  27. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  28. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  29. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  30. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  31. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 151
  32. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  33. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  34. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  35. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 79
  36. 2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
  37. 2021Reordering Examples Helps during Priming-based Few-Shot Learningcited by 50
  38. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 20; 2025: 20; 2026: 14.

63 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 909
  2. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  3. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  4. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
  5. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  6. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  7. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  8. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  9. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 51
  10. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 36
  11. 2024T2SG: Traffic Topology Scene Graph for Topology Reasoning in Autonomous Drivingcited by 35
  12. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 35
38 more
  1. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 34
  2. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Datacited by 33
  3. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 33
  4. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  5. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  6. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Modelscited by 20
  7. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 16
  8. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 15
  9. 2024DG-PIC: Domain Generalized Point-In-Context Learning for Point Cloud Understandingcited by 13
  10. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  11. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  12. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learningcited by 11
  13. 2025External Knowledge Injection for CLIP-Based Class-Incremental Learningcited by 11
  14. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  15. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answeringcited by 10
  16. 2025The Coherence Trap: When MLLM-Crafted Narratives Exploit Manipulated Visual Contextscited by 8
  17. 2023Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Modelscited by 7
  18. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 7
  19. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigationcited by 7
  20. 2024Zero-Shot Long-Form Video Understanding through Screenplaycited by 6
  21. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  22. 2023Manipulating the Label Space for In-Context Classificationcited by 5
  23. 2023Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learningcited by 5
  24. 2024LIVE: Learnable In-Context Vector for Visual Question Answeringcited by 5
  25. 2026FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptationcited by 5
  26. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learningcited by 4
  27. 2025LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compressioncited by 4
  28. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Modelscited by 4
  29. 2025MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderationcited by 3
  30. 2025FMLGS: Fast Multilevel Language Embedded Gaussians for Part-level Interactive Agentscited by 3
  31. 2025SignAligner: Harmonizing Complementary Pose Modalities for Coherent Sign Language Generationcited by 3
  32. 2025The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Modelscited by 3
  33. 2024Towards Lifelong Few-Shot Customization of Text-to-Image Diffusioncited by 2
  34. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysiscited by 2
  35. 2026PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignmentcited by 2
  36. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 2
  37. 2024Exploring the Distinctiveness and Fidelity of the Descriptions Generated by Large Vision-Language Modelscited by 1
  38. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1

50 most cited of 63 All 63 (JSON)