mlsanityData to 14 Oct 2026

HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understanding

arXiv 2410.06777 ↗First posted 9 Oct 2024cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The significant advancements in visual understanding and instruction following from Multimodal Large Language Models (MLLMs) have opened up more possibilities for broader applications in diverse and universal human-centric scenarios. However, existing image-text data may not support the precise modality alignment and integration of multi-grained information, which is crucial for human-centric visual understanding. In this paper, we introduce HERM-Bench, a benchmark for evaluating the human-centric understanding capabilities of MLLMs. Our work reveals the limitations of existing MLLMs in understanding complex human-centric scenarios. To address these challenges, we present HERM-100K, a comprehensive dataset with multi-level human-centric annotations, aimed at enhancing MLLMs' training. Furthermore, we develop HERM-7B, a MLLM that leverages enhanced training data from HERM-100K. Evaluations on HERM-Bench demonstrate that HERM-7B significantly outperforms existing MLLMs across various human-centric dimensions, reflecting the current inadequacy of data annotations used in MLLM training for human-centric visual understanding. This research emphasizes the importance of specialized datasets and benchmarks in advancing the MLLMs' capabilities for human-centric understanding.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 87 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

68 of 87 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan
  2. 2023Merlin:Empowering Multimodal LLMs with Foresight Minds30 Nov
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 58

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Segment Anythingcited by 8,234
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  4. 2023PaLM 2 Technical Reportcited by 1,162
  5. 2021Grounded Language-Image Pre-trainingcited by 1,156
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  8. 2022Human Motion Diffusion Modelcited by 1,080
  9. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,071
  10. 2016Modeling Context in Referring Expressionscited by 1,063
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  13. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  14. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  15. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  16. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  17. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  18. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  19. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  20. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  21. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  22. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
  23. 2023Improving CLIP Training with Language Rewritescited by 217
  24. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 182
  25. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  26. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  27. 2018Instance-level Human Parsing via Part Grouping Networkcited by 134
  28. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  29. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 94
  30. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  31. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  32. 2018LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Imagescited by 88
  33. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 83
  34. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  35. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 80
  36. 2018Look into Person: Joint Body Parsing & Pose Estimation Network and A New Benchmarkcited by 79
  37. 2017AI Challenger : A Large-scale Dataset for Going Deeper in Image Understandingcited by 73
  38. 2023UniHCP: A Unified Model for Human-Centric Perceptionscited by 59

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  2. 2024HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarkscited by 11
  3. 2025Research-Oriented Human-Centric Evaluation for Foundation Modelscited by 5
  4. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  5. 2026MHPR: Multidimensional Human Perception and Reasoning Benchmark for Large Vision-Languate Modelscited by 1
  6. 2025QoNext: Towards Next-generation QoE for Foundation Modelscited by 0
  7. 2026Employing Vision-Language Models for Face Image Quality Assessmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models
  2. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
  3. 2025HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models