# The Future of MLLM Prompting is Adaptive: A Comprehensive Experimental Evaluation of Prompt Engineering Methods for Robust Multimodal Performance

arXiv: https://arxiv.org/abs/2504.10179 · HTML: https://mlsanity.com/paper/2504.10179/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 14 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Built on by (most cited first, 7 of 7)

- [Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting](https://mlsanity.com/paper/2508.02429/) (2025)
- [ArogyaSutra: A Multi-Agent Framework for Multimodal Medical Reasoning in Indic Languages](https://mlsanity.com/paper/2606.13572/) (2026)
- [UniAPO: Unified Multimodal Automated Prompt Optimization](https://mlsanity.com/paper/2508.17890/) (2025)
- [Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenization](https://mlsanity.com/paper/2509.05695/) (2025)
- [Exploring Diagnostic Prompting Approach for Multimodal LLM-based Visual Complexity Assessment: A Case Study of Amazon Search Result Pages](https://mlsanity.com/paper/2512.00082/) (2025)
- [Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metric](https://mlsanity.com/paper/2605.06201/) (2026)
- [Vision-Language Model Reasoning for Contextual Semantic Mapping in Intralogistics](https://mlsanity.com/paper/2606.24814/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) are set to transform how machines process and generate human-like responses by integrating diverse modalities such as text, images, and code. Yet, effectively harnessing their capabilities hinges on optimal prompt engineering. We present a comprehensive experimental evaluation of seven prompt engineering methods applied to 13 open-source MLLMs over 24 tasks spanning Reasoning and Compositionality, Multimodal Understanding and Alignment, Complex Code Generation and Execution, and Knowledge Retrieval and Integration. Our approach stratifies models by parameter count into Small (<4B), Medium (4B-10B), and Large (>10B) categories and compares prompting techniques including Zero-Shot, One-Shot, Few-Shot, Chain-of-Thought, Analogical, Generated Knowledge, and Tree-of-Thought. While Large MLLMs excel in structured tasks such as code generation, achieving accuracies up to 96.88% under Few-Shot prompting, all models struggle with complex reasoning and abstract understanding, often yielding accuracies below 60% and high hallucination rates. Structured reasoning prompts frequently increased hallucination up to 75% in small models and led to longer response times (over 20 seconds in Large MLLMs), while simpler prompting methods provided more concise and efficient outputs. No single prompting method uniformly optimises all task types. Instead, adaptive strategies combining example-based guidance with selective structured reasoning are essential to enhance robustness, efficiency, and factual accuracy. Our findings offer practical recommendations for prompt engineering and support more reliable deployment of MLLMs across applications including AI-assisted coding, knowledge retrieval, and multimodal content understanding.