mlsanityData to 14 Oct 2026

HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding

arXiv 2501.15111 ↗First posted 25 Jan 2025cs.CVcited by 60

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In human-centric scenes, the ability to simultaneously understand visual and auditory information is crucial. While recent omni models can process multiple modalities, they generally lack effectiveness in human-centric scenes due to the absence of large-scale, specialized datasets and non-targeted architectures. In this work, we developed HumanOmni, the industry's first human-centric Omni-multimodal large language model. We constructed a dataset containing over 2.4 million human-centric video clips with detailed captions and more than 14 million instructions, facilitating the understanding of diverse human-centric scenes. HumanOmni includes three specialized branches for understanding different types of scenes. It adaptively fuses features from these branches based on user instructions, significantly enhancing visual understanding in scenes centered around individuals. Moreover, HumanOmni integrates audio features to ensure a comprehensive understanding of environments and individuals. Our experiments validate HumanOmni's advanced capabilities in handling human-centric scenes across a variety of tasks, including emotion recognition, facial expression description, and action understanding. Our model will be open-sourced to facilitate further development and collaboration within both academia and industry.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

51 of 60 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 857
38 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  3. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  5. 2024Qwen2-Audio Technical Reportcited by 560
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  7. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  8. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  9. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  10. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  11. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  12. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  13. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  14. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  15. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  16. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  17. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  18. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  19. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  20. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  21. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  22. 2020DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wildcited by 128
  23. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  24. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  25. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 121
  26. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 109
  27. 2019Context-Aware Emotion Recognition Networkscited by 99
  28. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  29. 2022FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videoscited by 69
  30. 2022MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wildcited by 65
  31. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61
  32. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  33. 2023Prompting Visual-Language Models for Dynamic Facial Expression Recognitioncited by 44
  34. 2023MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognitioncited by 43
  35. 2024Baichuan-Omni Technical Reportcited by 42
  36. 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
  37. 2025Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysiscited by 26
  38. 2025LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understandingcited by 10

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 37.

60 papers.

Sort:

By area:

  1. 2025R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learningcited by 100
  2. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  3. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 55
  4. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
  5. 2025LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoningcited by 27
  6. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 26
  7. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Surveycited by 19
  8. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  9. 2025FaceLLM: A Multimodal Large Language Model for Face Understandingcited by 15
  10. 2025Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoningcited by 12
  11. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 12
  12. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 11
38 more
  1. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  2. 2025ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understandingcited by 7
  3. 2025FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPOcited by 6
  4. 2025Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understandingcited by 6
  5. 2025VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Modelscited by 5
  6. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understandingcited by 5
  7. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  8. 2026GeoAgent: Learning to Geolocate Everywhere with Reinforced Geographic Characteristicscited by 5
  9. 2025VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Modelscited by 4
  10. 2026EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessmentcited by 4
  11. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 4
  12. 2026SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaborationcited by 4
  13. 2025Can Large Language Models Capture Video Game Engagement?cited by 3
  14. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Promptingcited by 3
  15. 2026HumanOmni-Speaker: Identifying Who said What and Whencited by 3
  16. 2026Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoningcited by 3
  17. 2026AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compressioncited by 3
  18. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  19. 2026Not all Blends are Equal: The BLEMORE Dataset of Blended Emotion Expressions with Relative Salience Annotationscited by 2
  20. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 2
  21. 2026Multi-Dimensional Quality Assessment for AI-Generated Human-Centric Videos: Dataset and Modelcited by 2
  22. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1
  23. 2025Grounding Intelligence in Movementcited by 1
  24. 2025Uncertainty-Aware Collaborative System of Large and Small Models for Multimodal Sentiment Analysiscited by 1
  25. 2025Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Soundcited by 1
  26. 2026Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Datacited by 1
  27. 2026Do Vision Language Models Understand Human Engagement in Games?cited by 1
  28. 2026AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teacherscited by 1
  29. 2026Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspectivecited by 1
  30. 2026TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videoscited by 1
  31. 2026GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactionscited by 1
  32. 2026AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMscited by 1
  33. 2026E3mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignmentcited by 1
  34. 2026HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligencecited by 1
  35. 2026Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognitioncited by 0
  36. 2026TDMM-LM: Bridging Facial Understanding and Animation via Language Modelscited by 0
  37. 2026C2F-Thinker: Coarse-to-Fine Reasoning with Hint-Guided Reinforcement Learning for Multimodal Sentiment Analysiscited by 0
  38. 2026Watching Movies Like a Human: Egocentric Emotion Understanding for Embodied Companionscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarks
  2. 2025HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding
  3. 2025Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models
  4. 2026FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding
  5. 2026HumanNet: Scaling Human-centric Video Learning to One Million Hours
  6. 2024HumanVLM: Foundation for Human-Scene Vision-Language Model
  7. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
  8. 2024HourVideo: 1-Hour Video-Language Understanding