mlsanityData to 7 Oct 2026

HumanOmni-Speaker: Identifying Who said What and When

arXiv 2603.21664 ↗First posted 23 Mar 2026cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

While Omni-modal Large Language Models have made strides in joint sensory processing, they fundamentally struggle with a cornerstone of human interaction: deciphering complex, multi-person conversational dynamics to accurately answer ``Who said what and when.'' Current models suffer from an ``illusion of competence'' -- they exploit visual biases in conventional benchmarks to bypass genuine cross-modal alignment, while relying on sparse, low-frame-rate visual sampling that destroys crucial high-frequency dynamics like lip movements. To shatter this illusion, we introduce Visual-Registered Speaker Diarization and Recognition (VR-SDR) and the HumanOmni-Speaker Benchmark. By strictly eliminating visual shortcuts, this rigorous paradigm demands true end-to-end spatio-temporal identity binding using only natural language queries. To overcome the underlying architectural perception gap, we propose HumanOmni-Speaker, powered by a Visual Delta Encoder. By sampling raw video at 25 fps and explicitly compressing inter-frame motion residuals into just 6 tokens per frame, it captures fine-grained visemes and speaker trajectories without triggering a catastrophic token explosion. Ultimately, HumanOmni-Speaker demonstrates strong multimodal synergy, natively enabling end-to-end lip-reading and high-precision spatial localization without intrusive cropping, and achieving superior performance across a wide spectrum of speaker-centric tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

31 of 36 references.

References · 31

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  3. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  4. 2023Large Language Modelscited by 774
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  6. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  7. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  8. 2016Lip Reading Sentences in the Wildcited by 226
  9. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  10. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  11. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  12. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
19 more
  1. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
  2. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  3. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  4. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
  5. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  6. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 81
  7. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  8. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 61
  9. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  10. 2025Embodied AI Agents: Modeling the Worldcited by 57
  11. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  12. 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 35
  13. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  14. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  15. 2016Cross-modal Supervision for Learning Active Speaker Detection in Videocited by 21
  16. 2025SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Modelscited by 14
  17. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  18. 2024Unveiling Visual Biases in Audio-Visual Localization Benchmarkscited by 3
  19. 2025CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.