mlsanityData to 7 Oct 2026

Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences

arXiv 2401.10529 ↗First posted 19 Jan 2024cs.CV cs.AI cs.CLcited by 93

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated proficiency in handling a variety of visual-language tasks. However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated. To address this challenge, this paper introduces Mementos, a new benchmark designed to assess MLLMs' sequential image reasoning abilities. Mementos features 4,761 diverse image sequences with varying lengths. We also employ a GPT-4 assisted method to evaluate MLLM reasoning performance. Through a careful evaluation of nine recent MLLMs on Mementos, including GPT-4V and Gemini, we find that they struggle to accurately describe dynamic information about given image sequences, often leading to hallucinations/misrepresentations of objects and their corresponding behaviors. Our quantitative analysis and case studies identify three key factors impacting MLLMs' sequential image reasoning: the correlation between object and behavioral hallucinations, the influence of cooccurring behaviors, and the compounding impact of behavioral hallucinations. Our dataset is available at https://github.com/umd-huang-lab/Mementos.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 47 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

26 of 47 references.

References · 26

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  9. 2018Object Hallucination in Image Captioningcited by 661
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  12. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
14 more
  1. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  2. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  3. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  4. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  5. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  6. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  7. 2023Vision-Language Models as a Source of Rewardscited by 65
  8. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  9. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 52
  10. 2023Mitigating Hallucination in Visual Language Models with Visual Supervisioncited by 49
  11. 2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48
  12. 2023COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RLcited by 14
  13. 2023Scalable Prompt Generation for Semi-supervised Learning with Language Modelscited by 10
  14. 2023C-Disentanglement: Discovering Causally-Independent Generative Factors under an Inductive Bias of Confoundercited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 45; 2025: 37; 2026: 11.

93 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  3. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  5. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  6. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  7. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  8. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  9. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  10. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  11. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  12. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
38 more
  1. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  2. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  3. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  4. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  5. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  6. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 50
  7. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  8. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  9. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  10. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 42
  11. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  12. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  13. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 32
  14. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  15. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  16. 2024Automated Data Curation for Robust Language Model Fine-Tuningcited by 27
  17. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  18. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platformscited by 26
  19. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  20. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  21. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  22. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  23. 2024Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehensioncited by 23
  24. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibrationcited by 23
  25. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  26. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  27. 2024Large Language Model Benchmarks in Medical Taskscited by 16
  28. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  29. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  30. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucinationcited by 14
  31. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  32. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 14
  33. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEscited by 13
  34. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Modelscited by 13
  35. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generationcited by 13
  36. 2024Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarioscited by 12
  37. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  38. 2024HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embeddingcited by 10

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  2. 2023Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Models
  3. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
  4. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
  7. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies
  8. 2024Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination