mlsanityData to 7 Oct 2026

FewMMBench: A Benchmark for Multimodal Few-Shot Learning

arXiv 2602.21854 ↗First posted 25 Feb 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

As multimodal large language models (MLLMs) advance in handling interleaved image-text data, assessing their few-shot learning capabilities remains an open challenge. In this paper, we introduce FewMMBench, a comprehensive benchmark designed to evaluate MLLMs under few-shot conditions, with a focus on In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting. Covering a diverse suite of multimodal understanding tasks, from attribute recognition to temporal reasoning, FewMMBench enables systematic analysis across task types, model families, and prompting strategies. We evaluate 26 open-weight MLLMs from six model families across zero-shot, few-shot, and CoT-augmented few-shot settings. Our findings reveal that instruction-tuned models exhibit strong zero-shot performance but benefit minimally, or even regress, with additional demonstrations or CoT reasoning. Retrieval-based demonstrations and increased context size also yield limited gains. These results highlight FewMMBench as a rigorous testbed for diagnosing and advancing few-shot capabilities in multimodal LLMs. The data is available at: https://huggingface.co/datasets/mustafaa/FewMMBench

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 41 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

24 of 41 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs3 Mar
  2. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities1 Aug
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  6. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs17 Jun
  7. 2024MileBench: Benchmarking MLLMs in Long Context29 Apr
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  9. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  11. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  8. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  9. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  10. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  11. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
  12. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
1 more
  1. 2025Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shotcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
  2. 2023On the Performance of Multimodal Language Models
  3. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  4. 2026Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learners
  5. 2024Many-Shot In-Context Learning
  6. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
  7. 2023Link-Context Learning for Multimodal LLMs
  8. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training