mlsanityData to 14 Oct 2026

Do MLLMs Exhibit Human-like Perceptual Behaviors? HVSBench: A Benchmark for MLLM Alignment with Human Perceptual Behavior

arXiv 2412.09603 ↗First posted 12 Dec 2024cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) excel at many vision tasks, it is unknown if they exhibit human-like perceptual behaviors. To evaluate this, we introduce HVSBench, the first large-scale benchmark with over 85,000 samples designed to test MLLM alignment with the human visual system (HVS). The benchmark covers 13 categories across 5 key fields: Prominence, Subitizing, Prioritizing, Free-Viewing, and Searching. Our comprehensive evaluation reveals a significant perceptual gap: even state-of-the-art MLLMs achieve only moderate results. In contrast, human participants demonstrate strong performance, significantly outperforming all models. This underscores the high quality of HVSBench and the need for more human-aligned AI. We believe our benchmark will be a critical tool for developing the next generation of explainable MLLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 56 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

32 of 56 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  10. 2019Towards VQA Models That Can Readcited by 1,718
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
14 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  2. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  4. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  5. 2014The Secrets of Salient Object Segmentationcited by 200
  6. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  7. 2016Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Modelcited by 91
  8. 2020RGB-D Salient Object Detection: A Surveycited by 73
  9. 2016DeepGaze II: Reading fixations from deep features trained on object recognitioncited by 55
  10. 2020Predicting Goal-directed Human Attention Using Inverse Reinforcement Learningcited by 53
  11. 2023Unifying Top-down and Bottom-up Scanpath Prediction Using Transformerscited by 22
  12. 2016Salient Object Subitizingcited by 13
  13. 2024CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusioncited by 12
  14. 2014What is a salient object? A dataset and a baseline model for salient object detectioncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025GazeLLM: Multimodal LLMs incorporating Human Visual Attentioncited by 9
  2. 2026Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humanscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?
  2. 2025MANBench: Is Your Multimodal Model Smarter than Human?
  3. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
  4. 2025VisualActBench: Can VLMs See and Act like a Human?
  5. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
  6. 2025From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
  7. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
  8. 2025HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding