mlsanityData to 14 Oct 2026

InterFeedback: Unveiling Interactive Intelligence of Large Multimodal Models via Human Feedback

arXiv 2502.15027 ↗First posted 20 Feb 2025cs.CL cs.AI cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Existing benchmarks do not test Large Multimodal Models (LMMs) on their interactive intelligence with human users, which is vital for developing general-purpose AI assistants. We design InterFeedback, an interactive framework, which can be applied to any LMM and dataset to assess this ability autonomously. On top of this, we introduce InterFeedback-Bench which evaluates interactive intelligence using two representative datasets, MMMU-Pro and MathVerse, to test 10 different open-source LMMs. Additionally, we present InterFeedback-Human, a newly collected dataset of 120 cases designed for manually testing interactive performance in leading models such as OpenAI-o1 and Claude-Sonnet-4. Our evaluation results indicate that even the state-of-the-art LMM, OpenAI-o1, struggles to refine its responses based on human feedback, achieving an average score of less than 50%. Our findings point to the need for methods that can enhance LMMs' capabilities to interpret and benefit from feedback.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 62 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

40 of 62 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  5. 2023Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator11 Dec
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 31

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
19 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  3. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  4. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  5. 2018nocaps: novel object captioning at scalecited by 444
  6. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  7. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  8. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  9. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
  10. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
  11. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 48
  12. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  13. 2024ChatShop: Interactive Information Seeking with Language Agentscited by 24
  14. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24
  15. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  16. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18
  17. 2023Human-in-the-Loop Schema Inductioncited by 13
  18. 2024FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Modelscited by 6
  19. 2023Towards the design of user-centric strategy recommendation systems for collaborative Human-AI taskscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TempCloze: Can Video-LLMs Identify the Missing Middle?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models
  2. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback
  3. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback
  4. 2025MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
  5. 2025HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs
  6. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
  7. 2025ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks
  8. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning