mlsanityData to 7 Oct 2026

Gamified crowd-sourcing of high-quality data for visual fine-tuning

arXiv 2410.04038 ↗First posted 5 Oct 2024cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

This paper introduces Gamified Adversarial Prompting (GAP), a framework that crowd-sources high-quality data for visual instruction tuning of large multimodal models. GAP transforms the data collection process into an engaging game, incentivizing players to provide fine-grained, challenging questions and answers that target gaps in the model's knowledge. Our contributions include (1) an approach to capture question-answer pairs from humans that directly address weaknesses in a model's knowledge, (2) a method for evaluating and rewarding players that successfully incentivizes them to provide high-quality submissions, and (3) a scalable, gamified platform that succeeds in collecting this data from over 50,000 participants in just a few weeks. Our implementation of GAP has significantly improved the accuracy of a small multimodal model, namely MiniCPM-Llama3-V-2.5-8B, increasing its GPT score from 0.147 to 0.477 on our dataset, approaching the benchmark set by the much larger GPT-4V. Moreover, we demonstrate that the data generated using MiniCPM-Llama3-V-2.5-8B also enhances its performance across other benchmarks, and exhibits cross-model benefits. Specifically, the same data improves the performance of QWEN2-VL-2B and QWEN2-VL-7B on the same multiple benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 61 references that are in the map. First posted on arXiv 5 Oct 2024.

Builds on

47 of 61 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
35 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  12. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  13. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  14. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  15. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  16. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 350
  17. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  18. 2024What matters when building vision-language models?cited by 321
  19. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  20. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  21. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  22. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  23. 2020Adversarial Filters of Dataset Biasescited by 159
  24. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  25. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  26. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  27. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  28. 2019Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networkscited by 112
  29. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  30. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  31. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  32. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  33. 2021Fool Me Twice: Entailment from Wikipedia Gamificationcited by 50
  34. 2018On the effectiveness of task granularity for transfer learningcited by 35
  35. 2022Label Sleuth: From Unlabeled Text to a Classifier in a Few Hourscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GAMBIT: A Gamified Jailbreak Framework for Multimodal Large Language Models
  2. 2026PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence
  3. 2025VideoGameBench: Can Vision-Language Models complete popular video games?
  4. 2025PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solving
  5. 2023GlitchBench: Can large multimodal models detect video game glitches?
  6. 2024VideoGameBunny: Towards vision assistants for video games
  7. 2022WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models
  8. 2024GameLabel-10K: Collecting Image Preference Data Through Mobile Game Crowdsourcing