mlsanityData to 7 Oct 2026

Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding

arXiv 2407.08150 ↗First posted 11 Jul 2024cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

Understanding of video creativity and content often varies among individuals, with differences in focal points and cognitive levels across different ages, experiences, and genders. There is currently a lack of research in this area, and most existing benchmarks suffer from several drawbacks: 1) a limited number of modalities and answers with restrictive length; 2) the content and scenarios within the videos are excessively monotonous, transmitting allegories and emotions that are overly simplistic. To bridge the gap to real-world applications, we introduce a large-scale Subjective Response Indicators for Advertisement Videos dataset, namely SRI-ADV. Specifically, we collected real changes in Electroencephalographic (EEG) and eye-tracking regions from different demographics while they viewed identical video content. Utilizing this multi-modal dataset, we developed tasks and protocols to analyze and evaluate the extent of cognitive understanding of video content among different users. Along with the dataset, we designed a Hypergraph Multi-modal Large Language Model (HMLLM) to explore the associations among different demographics, video elements, EEG, and eye-tracking indicators. HMLLM could bridge semantic gaps across rich modalities and integrate information beyond different modalities to perform logical reasoning. Extensive experimental evaluations on SRI-ADV and other additional video-based generative performance benchmarks demonstrate the effectiveness of our method. The codes and dataset will be released at https://github.com/mininglamp-MLLM/HMLLM.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 90 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

50 of 90 references.

References · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2018SlowFast Networks for Video Recognitioncited by 1,862
38 more
  1. 2021ViViT: A Video Vision Transformercited by 1,610
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  8. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  13. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  14. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  15. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  16. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  17. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  18. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  19. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  20. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  21. 2018Hypergraph Neural Networkscited by 482
  22. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  23. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  24. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  25. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  26. 2021Video Transformer Networkcited by 271
  27. 2017Temporal Action Detection with Structured Segment Networkscited by 265
  28. 2019Deep learning-based electroencephalography analysis: a systematic reviewcited by 155
  29. 2021UniGNN: a Unified Framework for Graph and Hypergraph Neural Networkscited by 127
  30. 2017Single Shot Temporal Action Detectioncited by 123
  31. 2020HyperSAGE: Generalizing Inductive Representation Learning on Hypergraphscited by 53
  32. 2018VideoMem: Constructing, Analyzing, Predicting Short-term and Long-term Video Memorabilitycited by 24
  33. 2020Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorabilitycited by 23
  34. 2015Learning Hypergraph-regularized Attribute Predictorscited by 13
  35. 2016Elastic Net Hypergraph Learning for Image Clustering and Semi-supervised Classificationcited by 9
  36. 2023Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behaviorcited by 9
  37. 2020Investigating EEG-Based Functional Connectivity Patterns for Multimodal Emotion Recognitioncited by 4
  38. 2021Multi-modal Ensemble Models for Predicting Video Memorabilitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Large Language Models for EEG: A Comprehensive Survey and Taxonomycited by 4
  2. 2025Towards Consumer-Grade Cybersickness Prediction: Multi-Model Alignment for Real-Time Vision-Only Inferencecited by 1
  3. 2025PRE-MAP: Personalized Reinforced Eye-tracking Multimodal LLM for High-Resolution Multi-Attribute Point Predictioncited by 0
  4. 2026OZ-TAL: Online Zero-Shot Temporal Action Localizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Dataset Generation Scheme Based on Video2EEG-SPGN-Diffusion for SEED-VD
  2. 2026Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation
  3. 2026Human-Centered Multimodal Fusion for Sexism Detection in Memes with Eye-Tracking, Heart Rate, and EEG Signals
  4. 2025EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment
  5. 2025Seeing is Not Understanding: A Benchmark on Perception-Cognition Disparities in Large Language Models
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
  7. 2025VideoMind: An Omni-Modal Video Dataset with Intent Grounding for Deep-Cognitive Video Understanding
  8. 2025Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts