mlsanityData to 7 Oct 2026

Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

arXiv 2510.01284 ↗First posted 30 Sep 2025cs.MM cs.CV cs.SDcited by 76

Not on the map. Why?

Abstract (arXiv)

Audio-video generation has often relied on complex multi-stage architectures or sequential synthesis of sound and visuals. We introduce Ovi, a unified paradigm for audio-video generation that models the two modalities as a single generative process. By using blockwise cross-modal fusion of twin-DiT modules, Ovi achieves natural synchronization and removes the need for separate pipelines or post hoc alignment. To facilitate fine-grained multimodal fusion modeling, we initialize an audio tower with an architecture identical to that of a strong pretrained video model. Trained from scratch on hundreds of thousands of hours of raw audio, the audio tower learns to generate realistic sound effects, as well as speech that conveys rich speaker identity and emotion. Fusion is obtained by jointly training the identical video and audio towers via blockwise exchange of timing (via scaled-RoPE embeddings) and semantics (through bidirectional cross-attention) on a vast video corpus. Our model enables cinematic storytelling with natural speech and accurate, context-matched sound effects, producing movie-grade video clips. All the demos, code and model weights are published at https://aaxwaz.github.io/Ovi

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 37 of its 42 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

37 of 42 references.

References · 37

  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  7. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  8. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  9. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  10. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  11. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  12. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
25 more
  1. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  2. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
  3. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  4. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  5. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  6. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  7. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  8. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  9. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  10. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
  11. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  12. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  13. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  14. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  15. 2025HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characterscited by 64
  16. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
  17. 2025ACE-Step: A Step Towards Music Generation Foundation Modelcited by 54
  18. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 52
  19. 2024Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesiscited by 45
  20. 2025HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioningcited by 44
  21. 2024Taming Data and Transformers for Audio Generationcited by 28
  22. 2025TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Modelscited by 26
  23. 2025MagicInfinite: Generating Infinite Talking Videos with Your Words and Voicecited by 16
  24. 2024Semantically consistent Video-to-Audio Generation using Multimodal Language Large Modelcited by 11
  25. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 68.

76 papers.

Sort:

By area:

  1. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
  2. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  3. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  4. 2026Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Modelcited by 23
  5. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  6. 2025T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generationcited by 19
  7. 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 19
  8. 2026Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillationcited by 13
  9. 2026SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing modelcited by 11
  10. 2025PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generationcited by 10
  11. 2025Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interactioncited by 9
  12. 2026RoboStereo: Dual-Tower 4D Embodied World Models for Unified Policy Optimizationcited by 9
38 more
  1. 2025AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refinercited by 8
  2. 2026Apollo: Unified Multi-Task Audio-Video Joint Generationcited by 8
  3. 2026Do Joint Audio-Video Generation Models Understand Physics?cited by 8
  4. 2026MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloningcited by 7
  5. 2026OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generationcited by 7
  6. 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Groundingcited by 6
  7. 2026Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generationcited by 6
  8. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 6
  9. 2026ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRAcited by 5
  10. 2026CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generationcited by 5
  11. 20253MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generationcited by 4
  12. 2025MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generationcited by 4
  13. 2026Improving Joint Audio-Video Generation with Cross-Modal Context Learningcited by 4
  14. 2026MaineCoon: Pursuing A Real-Time Audio-Visual Social World Modelcited by 4
  15. 2026UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gatingcited by 4
  16. 2026Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?cited by 3
  17. 2026OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Modelcited by 3
  18. 2026CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentratingcited by 3
  19. 2026AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Modelscited by 3
  20. 2026Baton: Explicit Semantic Blueprints for Joint Video-Audio Generationcited by 3
  21. 2026StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestrationcited by 3
  22. 2026MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generationcited by 3
  23. 2026Inference-Time Scaling for Joint Audio-Video Generationcited by 3
  24. 2026VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generationcited by 3
  25. 2026UniTalking: A Unified Audio-Video Framework for Talking Portrait Generationcited by 2
  26. 2026Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modelingcited by 2
  27. 2026Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generationcited by 2
  28. 2026TIDE: Task-Isolated Diffusion for Unified Video Editing and Generationcited by 2
  29. 2026InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videoscited by 2
  30. 2026Identity as Presence: Towards Appearance and Voice Personalized Joint Audio-Video Generationcited by 1
  31. 2026InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidancecited by 1
  32. 2026OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generationcited by 1
  33. 2026Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generationcited by 1
  34. 2026SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learningcited by 1
  35. 2026AVTok: 1D Unified Tokenization for Holistic Audio-Video Generationcited by 1
  36. 2026Cycle-World: Mitigating Error Accumulation in Long-term Video World Models via Reverse-Prediction Cycle Consistencycited by 1
  37. 2026Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Video Avatar Generationcited by 1
  38. 2026Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attentioncited by 1

50 most cited of 76 All 76 (JSON)