HumanOmni-Speaker: Identifying Who said What and When
Not on the map. Why?
Abstract (arXiv)
While Omni-modal Large Language Models have made strides in joint sensory processing, they fundamentally struggle with a cornerstone of human interaction: deciphering complex, multi-person conversational dynamics to accurately answer ``Who said what and when.'' Current models suffer from an ``illusion of competence'' -- they exploit visual biases in conventional benchmarks to bypass genuine cross-modal alignment, while relying on sparse, low-frame-rate visual sampling that destroys crucial high-frequency dynamics like lip movements. To shatter this illusion, we introduce Visual-Registered Speaker Diarization and Recognition (VR-SDR) and the HumanOmni-Speaker Benchmark. By strictly eliminating visual shortcuts, this rigorous paradigm demands true end-to-end spatio-temporal identity binding using only natural language queries. To overcome the underlying architectural perception gap, we propose HumanOmni-Speaker, powered by a Visual Delta Encoder. By sampling raw video at 25 fps and explicitly compressing inter-frame motion residuals into just 6 tokens per frame, it captures fine-grained visemes and speaker trajectories without triggering a catastrophic token explosion. Ultimately, HumanOmni-Speaker demonstrates strong multimodal synergy, natively enabling end-to-end lip-reading and high-precision spatial localization without intrusive cropping, and achieving superior performance across a wide spectrum of speaker-centric tasks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 23 Mar 2026.
Builds on
31 of 36 references.
References · 31
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2014Show and Tell: A Neural Image Caption Generatorcited by 877
- 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
- 2023Large Language Modelscited by 774
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
- 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
- 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
- 2016Lip Reading Sentences in the Wildcited by 226
- 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
- 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
- 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
- 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
19 more
- 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
- 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
- 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
- 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
- 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
- 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 81
- 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
- 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 61
- 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
- 2025Embodied AI Agents: Modeling the Worldcited by 57
- 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
- 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 35
- 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
- 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
- 2016Cross-modal Supervision for Learning Active Speaker Detection in Videocited by 21
- 2025SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Modelscited by 14
- 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
- 2024Unveiling Visual Biases in Audio-Visual Localization Benchmarkscited by 3
- 2025CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronizationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.