mlsanityData to 7 Oct 2026

Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning

arXiv 2407.12498 ↗First posted 17 Jul 2024cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The linguistic capabilities of Multimodal Large Language Models (MLLMs) are critical for their effective application across diverse tasks. This study aims to evaluate the performance of MLLMs on the VALSE benchmark, focusing on the efficacy of few-shot In-Context Learning (ICL), and Chain-of-Thought (CoT) prompting. We conducted a comprehensive assessment of state-of-the-art MLLMs, varying in model size and pretraining datasets. The experimental results reveal that ICL and CoT prompting significantly boost model performance, particularly in tasks requiring complex reasoning and contextual understanding. Models pretrained on captioning datasets show superior zero-shot performance, while those trained on interleaved image-text data benefit from few-shot learning. Our findings provide valuable insights into optimizing MLLMs for better grounding of language in visual contexts, highlighting the importance of the composition of pretraining data and the potential of few-shot learning strategies to improve the reasoning abilities of MLLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 87 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

51 of 87 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  4. 2023Generative Multimodal Models are In-Context Learners20 Dec
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning1 Oct
  9. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
28 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  9. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  10. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  11. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  12. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  13. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  14. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  15. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  16. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  17. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  18. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  19. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  20. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  21. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 71
  22. 2022All You May Need for VQA are Image Captionscited by 65
  23. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 59
  24. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  25. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  26. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  27. 2020Multimodal Dialogue State Tracking By QA Approach with Data Augmentationcited by 7
  28. 2020Data augmentation techniques for the Video Question Answering taskcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FewMMBench: A Benchmark for Multimodal Few-Shot Learning
  2. 2024Can MLLMs Perform Text-to-Image In-Context Learning?
  3. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  4. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
  5. 2023On the Performance of Multimodal Language Models
  6. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  7. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  8. 2025Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognition