mlsanityData to 7 Oct 2026

FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models

arXiv 2407.11522 ↗First posted 16 Jul 2024cs.CVcited by 6

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision language models (VLMs) have achieved impressive progress in diverse applications, becoming a prevalent research direction. In this paper, we build FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn conversations that are derived from 27 source datasets, empowering VLMs to spontaneously refine their responses based on user feedback across diverse tasks. To scale up the data collection, FIRE is collected in two components: FIRE-100K and FIRE-1M, where FIRE-100K is generated by GPT-4V, and FIRE-1M is freely generated via models trained on FIRE-100K. Then, we build FIRE-Bench, a benchmark to comprehensively evaluate the feedback-refining capability of VLMs, which contains 11K feedback-refinement conversations as the test data, two evaluation settings, and a model to provide feedback for VLMs. We develop the FIRE-LLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, which shows remarkable feedback-refining capability on FIRE-Bench and outperforms untrained VLMs by 50%, making more efficient user-agent interactions and underscoring the significance of the FIRE dataset.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 72 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

64 of 72 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?9 Apr
  2. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  3. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
  4. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  5. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback16 Nov
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023Mistral 7Bcited by 3,065
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  13. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  14. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  15. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  16. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  17. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  18. 2016A Diagram Is Worth A Dozen Imagescited by 793
  19. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  20. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  21. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  22. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  23. 2021OCR-free Document Understanding Transformercited by 400
  24. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  25. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  26. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  27. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  28. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  29. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  30. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  31. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  32. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  33. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  34. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  35. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  36. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
  37. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  38. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  2. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Modelscited by 13
  3. 2025CADReview: Automatically Reviewing CAD Programs with Error Detection and Correctioncited by 3
  4. 2026Building a Precise Video Language with Human-AI Oversightcited by 3
  5. 2026Bridging Modality Disconnect in Self-Reflection via Closed-Loop Visually Grounded Verificationcited by 2
  6. 2025InterFeedback: Unveiling Interactive Intelligence of Large Multimodal Models via Human Feedbackcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024What matters when building vision-language models?
  2. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
  3. 2026MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
  4. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment
  5. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  6. 2025CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models
  7. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Models
  8. 2025FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation