mlsanityData to 7 Oct 2026

MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation

arXiv 2502.11903 ↗First posted 17 Feb 2025cs.CLcited by 28

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent multimodal large language models (MLLMs) have demonstrated significant potential in open-ended conversation, generating more accurate and personalized responses. However, their abilities to memorize, recall, and reason in sustained interactions within real-world scenarios remain underexplored. This paper introduces MMRC, a Multi-Modal Real-world Conversation benchmark for evaluating six core open-ended abilities of MLLMs: information extraction, multi-turn reasoning, information update, image management, memory recall, and answer refusal. With data collected from real-world scenarios, MMRC comprises 5,120 conversations and 28,720 corresponding manually labeled questions, posing a significant challenge to existing MLLMs. Evaluations on 20 MLLMs in MMRC indicate an accuracy drop during open-ended interactions. We identify four common failure patterns: long-term memory degradation, inadequacies in updating factual knowledge, accumulated assumption of error propagation, and reluctance to say no. To mitigate these issues, we propose a simple yet effective NOTE-TAKING strategy, which can record key information from the conversation and remind the model during its responses, enhancing conversational capabilities. Experiments across six MLLMs demonstrate significant performance improvements.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 96 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

46 of 96 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs22 Nov
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  5. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images21 Jul
  6. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs17 Jun
  7. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  8. 2023VILA: On Pre-training for Visual Language Models12 Dec
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 36

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2024A Survey on LLM-as-a-Judgecited by 1,384
  7. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  8. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  9. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  10. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  11. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  12. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
24 more
  1. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
  2. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  3. 2020Recent Advances and Challenges in Task-oriented Dialog Systemcited by 78
  4. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  5. 2022Stepwise Feature Fusion: Local Guides Globalcited by 56
  6. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  7. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
  8. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  9. 2019What Makes a Good Conversation? Challenges in Designing Truly Conversational Agentscited by 42
  10. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  11. 2024MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMscited by 32
  12. 2023ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Eventscited by 31
  13. 2024OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understandingcited by 31
  14. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  15. 2018Dialog-context aware end-to-end speech recognitioncited by 27
  16. 2022DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentationcited by 22
  17. 2023Ethical Considerations for Responsible Data Curationcited by 17
  18. 2025Can Large Language Models Understand Preferences in Personalized Recommendation?cited by 8
  19. 2024Are Human Conversations Special? A Large Language Model Perspectivecited by 5
  20. 2024Goal Inference from Open-Ended Dialogcited by 4
  21. 2024Neighbor Does Matter: Density-Aware Contrastive Learning for Medical Semi-supervised Segmentationcited by 4
  22. 2023CLAID: Closing the Loop on AI & Data Collection -- A Cross-Platform Transparent Computing Middleware Framework for Smart Edge-Cloud and Digital Biomarker Applicationscited by 3
  23. 2024Discriminating retinal microvascular and neuronal differences related to migraines: Deep Learning based Crossectional Studycited by 3
  24. 2024Meta Curvature-Aware Minimization for Domain Generalizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 19.

28 papers.

Sort:

By area:

  1. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 49
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  3. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  4. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  5. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  6. 2025Towards Next-Generation Recommender Systems: A Benchmark for Personalized Recommendation Assistant with LLMscited by 20
  7. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videoscited by 12
  8. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  9. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 11
  10. 2025Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perceptioncited by 10
  11. 2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 6
  12. 2026SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memorycited by 5
16 more
  1. 2025AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understandingcited by 4
  2. 2026SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Modelscited by 3
  3. 2026FileGram: Grounding Agent Personalization in File-System Behavioral Tracescited by 3
  4. 2026WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interactioncited by 3
  5. 2026M3Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactionscited by 2
  6. 2026Memory Bear AI Memory Science Engine for Multimodal Affective Intelligence: A Technical Reportcited by 1
  7. 2026TF-SSD: A Strong Pipeline via Synergic Mask Filter for Training-free Co-salient Object Detectioncited by 1
  8. 2026TALENT: Target-aware Efficient Tuning for Referring Image Segmentationcited by 1
  9. 2026Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agentscited by 1
  10. 2026Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactionscited by 0
  11. 2026Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactionscited by 0
  12. 2026When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMscited by 0
  13. 2026SciReC: Diagnostic Evaluation of Multimodal, Multi-Turn Relational Reasoning with Adaptive Interactioncited by 0
  14. 2026AtomRec: Evolving Atomic Memory for Agentic Recommendationcited by 0
  15. 2026Re2A: Situated Conversational Recommendation via Rubric-based Preference Reasoning and Alignmentcited by 0
  16. 2026CUE-Mem: Benchmarking Long-Term User Memory via Implicit Cues in Multimodal Conversationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
  2. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models
  3. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  4. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  5. 2026MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models
  6. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models
  7. 2026Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings
  8. 2024Efficient Multimodal Large Language Models: A Survey