mlsanityData to 14 Oct 2026

HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models

arXiv 2506.03922 ↗First posted 4 Jun 2025cs.CL cs.AI cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated significant potential to advance a broad range of domains. However, current benchmarks for evaluating MLLMs primarily emphasize general knowledge and vertical step-by-step reasoning typical of STEM disciplines, while overlooking the distinct needs and potential of the Humanities and Social Sciences (HSS). Tasks in the HSS domain require more horizontal, interdisciplinary thinking and a deep integration of knowledge across related fields, which presents unique challenges for MLLMs, particularly in linking abstract concepts with corresponding visual representations. Addressing this gap, we present HSSBench, a dedicated benchmark designed to assess the capabilities of MLLMs on HSS tasks in multiple languages, including the six official languages of the United Nations. We also introduce a novel data generation pipeline tailored for HSS scenarios, in which multiple domain experts and automated agents collaborate to generate and iteratively refine each sample. HSSBench contains over 13,000 meticulously designed samples, covering six key categories. We benchmark more than 20 mainstream MLLMs on HSSBench and demonstrate that it poses significant challenges even for state-of-the-art models. We hope that this benchmark will inspire further research into enhancing the cross-disciplinary reasoning abilities of MLLMs, especially their capacity to internalize and connect knowledge across fields.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 85 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

45 of 85 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning21 May
  2. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection17 Feb
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  6. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  8. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts20 Feb
  9. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  11. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
22 more
  1. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
  2. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  3. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  4. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 278
  5. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  6. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  7. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  8. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 120
  9. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 120
  10. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  11. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 69
  12. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 68
  13. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  14. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 51
  15. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 47
  16. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
  17. 2025SepPrune: Structured Pruning for Efficient Deep Speech Separationcited by 31
  18. 2023Towards LLM-driven Dialogue State Trackingcited by 25
  19. 2024TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidationcited by 18
  20. 2024Generative Regression Based Watch Time Prediction for Short-Video Recommendationcited by 13
  21. 2024The Instinctive Bias: Spurious Images lead to Illusion in MLLMscited by 11
  22. 2024VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 12.

17 papers.

Sort:

By area:

  1. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 20
  2. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 18
  3. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
  4. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 12
  5. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  6. 2026From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoningcited by 8
  7. 2025KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AIcited by 7
  8. 2026Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicinecited by 4
  9. 2026Learning Cross-View Object Correspondence via Cycle-Consistent Mask Predictioncited by 3
  10. 2026Intrinsic Lorentz Neural Networkcited by 3
  11. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  12. 2026Visual Prompt-Agnostic Evolutioncited by 2
5 more
  1. 2026MultiPress: A Multi-Agent Framework for Interpretable Multimodal News Classificationcited by 2
  2. 2026DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Drivingcited by 2
  3. 2026Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBenchcited by 1
  4. 2026Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Modelscited by 0
  5. 2026SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
  2. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  3. 2026WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
  4. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  5. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  6. 2024A Survey on Benchmarks of Multimodal Large Language Models
  7. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMs
  8. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams