mlsanityData to 7 Oct 2026

JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization

arXiv 2503.23377 ↗First posted 30 Mar 2025cs.CV cs.AI cs.SDcited by 84

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion Transformer designed for synchronized audio-video generation (JAVG). Based on the powerful Diffusion Transformer (DiT) architecture, JavisDiT simultaneously generates high-quality audio and video content from open-ended user prompts in a unified framework. To ensure audio-video synchronization, we introduce a fine-grained spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and fine-grained spatio-temporal priors, guiding the synchronization between the visual and auditory components. Furthermore, we propose a new benchmark, JavisBench, which consists of 10,140 high-quality text-captioned sounding videos and focuses on synchronization evaluation in diverse and complex real-world scenarios. Further, we specifically devise a robust metric for measuring the synchrony between generated audio-video pairs in real-world content. Experimental results demonstrate that JavisDiT significantly outperforms existing methods by ensuring both high-quality generation and precise synchronization, setting a new standard for JAVG tasks. Our code, model, and data are available at https://javisverse.github.io/JavisDiT-page/.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 62 of its 73 references that are in the map. First posted on arXiv 30 Mar 2025.

Builds on

62 of 73 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation6 Feb
  2. 2024MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation2 Oct
  3. 2024A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation26 Sep
  4. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation11 Jun
  5. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation28 May
  6. 2023The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion8 Sep
  7. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  8. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  9. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  10. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation19 Dec

Other references · 52

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Flow Matching for Generative Modelingcited by 5,426
  10. 2024Qwen2.5 Technical Reportcited by 4,713
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
38 more
  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  4. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  10. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  11. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  12. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  13. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  14. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  15. 2024Qwen2-Audio Technical Reportcited by 543
  16. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  17. 2019Clotho: An Audio Captioning Datasetcited by 354
  18. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  19. 2022UL2: Unifying Language Learning Paradigmscited by 292
  20. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  21. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  22. 2022Audio-Visual Segmentationcited by 148
  23. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  24. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  25. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  26. 2022Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusioncited by 94
  27. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 94
  28. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 68
  29. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  30. 2023Separate Anything You Describecited by 43
  31. 2022Sound-Guided Semantic Video Generationcited by 35
  32. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 33
  33. 2024Audio-Synchronized Visual Animationcited by 32
  34. 2021What is the ground truth? Reliability of multi-annotator data for audio taggingcited by 29
  35. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  36. 2024SyncFlow: Toward Temporally Aligned Joint Audio-Video Generation from Textcited by 24
  37. 2024STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignmentcited by 20
  38. 2023Fine-grained Audible Video Descriptioncited by 17

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 64.

84 papers.

Sort:

By area:

  1. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  2. 2025Controllable Video Generation: A Surveycited by 93
  3. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 76
  4. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  5. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  6. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 33
  7. 2025VABench: A Comprehensive Benchmark for Audio-Video Generationcited by 22
  8. 2025Region-Constraint In-Context Generation for Instructional Video Editingcited by 22
  9. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  10. 2025T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generationcited by 19
  11. 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 19
  12. 2026AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generationcited by 18
38 more
  1. 2025UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalistcited by 15
  2. 2026Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillationcited by 13
  3. 2025Object-AVEdit: An Object-level Audio-Visual Editing Modelcited by 12
  4. 2025JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generationcited by 12
  5. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 12
  6. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 11
  7. 2026SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing modelcited by 11
  8. 2025Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interactioncited by 9
  9. 2025SMRABooth: Subject and Motion Representation Alignment for Customized Video Generationcited by 9
  10. 2026Apollo: Unified Multi-Task Audio-Video Joint Generationcited by 8
  11. 2026Do Joint Audio-Video Generation Models Understand Physics?cited by 8
  12. 2025UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generationcited by 7
  13. 2026MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloningcited by 7
  14. 2026Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generationcited by 6
  15. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 6
  16. 2026Native Audio-Visual Alignment for Generationcited by 6
  17. 2026ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRAcited by 5
  18. 2026CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generationcited by 5
  19. 20253MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generationcited by 4
  20. 2025MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generationcited by 4
  21. 2026MMControl: Unified Multi-Modal Control for Joint Audio-Video Generationcited by 4
  22. 2026InstructAV2AV: Instruction-Guided Audio-Video Joint Editingcited by 4
  23. 2026Echo-Memory: A Controlled Study of Memory in Action World Modelscited by 4
  24. 2026MaineCoon: Pursuing A Real-Time Audio-Visual Social World Modelcited by 4
  25. 2026StreamAV-Bench: A Comprehensive Benchmark for Streaming Audio-Video Generationcited by 4
  26. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Taskscited by 3
  27. 2026Baton: Explicit Semantic Blueprints for Joint Video-Audio Generationcited by 3
  28. 2026StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestrationcited by 3
  29. 2026MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generationcited by 3
  30. 2026Inference-Time Scaling for Joint Audio-Video Generationcited by 3
  31. 2026VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generationcited by 3
  32. 2025Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generationcited by 2
  33. 2026UniTalking: A Unified Audio-Video Framework for Talking Portrait Generationcited by 2
  34. 2026Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modelingcited by 2
  35. 2026Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generationcited by 2
  36. 2026InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videoscited by 2
  37. 2026AUHead: Realistic Emotional Talking Head Generation via Action Units Controlcited by 1
  38. 2026NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spacescited by 1

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
  2. 2025ProAV-DiT: A Projected Latent Diffusion Transformer for Efficient Synchronized Audio-Video Generation
  3. 2026Diffusion Models for Joint Audio-Video Generation
  4. 2026Real-Time Joint Audio-Video Generation by Parallel Adapter Composition
  5. 2025Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformers
  6. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation
  7. 2026Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation
  8. 2025In-Context Audio Control of Video Diffusion Transformers