mlsanityData to 7 Oct 2026

LTX-2: Efficient Joint Audio-Visual Foundation Model

arXiv 2601.03233 ↗First posted 6 Jan 2026cs.CVcited by 213

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Recent text-to-video diffusion models can generate compelling video sequences, yet they remain silent -- missing the semantic, emotional, and atmospheric cues that audio provides. We introduce LTX-2, an open-source foundational model capable of generating high-quality, temporally synchronized audiovisual content in a unified manner. LTX-2 consists of an asymmetric dual-stream transformer with a 14B-parameter video stream and a 5B-parameter audio stream, coupled through bidirectional audio-video cross-attention layers with temporal positional embeddings and cross-modality AdaLN for shared timestep conditioning. This architecture enables efficient training and inference of a unified audiovisual model while allocating more capacity for video generation than audio generation. We employ a multilingual text encoder for broader prompt understanding and introduce a modality-aware classifier-free guidance (modality-CFG) mechanism for improved audiovisual alignment and controllability. Beyond generating speech, LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene -- complete with natural background and foley elements. In our evaluations, the model achieves state-of-the-art audiovisual quality and prompt adherence among open-source systems, while delivering results comparable to proprietary models at a fraction of their computational cost and inference time. All model weights and code are publicly released.

Key facts. Cited by 213 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 30 of its 31 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

30 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  2. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models29 Jun
  3. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 27

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  8. 2025Gemma 3 Technical Reportcited by 1,686
  9. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  11. 2023Vision Transformers Need Registerscited by 844
  12. 2024LTX-Video: Realtime Video Latent Diffusioncited by 549
15 more
  1. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  2. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  3. 2025Transfer between Modalities with MetaQueriescited by 231
  4. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  5. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  6. 2022Analyzing Transformers in Embedding Spacecited by 131
  7. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  8. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  9. 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 76
  10. 2025Wan-S2V: Audio-Driven Cinematic Video Generationcited by 60
  11. 2024Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registerscited by 19
  12. 2025Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captionscited by 13
  13. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generationcited by 10
  14. 2025Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interactioncited by 9
  15. 2025CAFA: a Controllable Automatic Foley Artistcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 213.

213 papers.

Sort:

By area:

  1. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  2. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memorycited by 86
  3. 2026Helios: Real Real-Time Long Video Generation Modelcited by 56
  4. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 50
  5. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  6. 2026Rethinking Video Generation Model for the Embodied Worldcited by 30
  7. 2026A Very Big Video Reasoning Suitecited by 29
  8. 2026Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Modelcited by 23
  9. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  10. 2026LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Modelscited by 20
  11. 2026DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generationcited by 19
  12. 2026VGGRPO: Towards World-Consistent Video Generation with 4D Latent Rewardcited by 19
38 more
  1. 2026AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generationcited by 18
  2. 2026D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Modelscited by 17
  3. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 13
  4. 2026WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPGcited by 13
  5. 2026Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillationcited by 13
  6. 2026Wonder: Video World Model Done Bettercited by 13
  7. 2026WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Conceptscited by 12
  8. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 12
  9. 2026Demystifying Video Reasoningcited by 12
  10. 2026OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generatorscited by 12
  11. 2026SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing modelcited by 11
  12. 2026PhyGround: Benchmarking Physical Reasoning in Generative World Modelscited by 10
  13. 2026SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformercited by 10
  14. 2026EchoWM: Open and Enterable Omnimodal World Modelscited by 10
  15. 2026PISCO: Precise Video Instance Insertion with Sparse Controlcited by 9
  16. 2026VOID: Video Object and Interaction Deletioncited by 9
  17. 2026SolarWM: Open Data and Scalable Training for Long-Horizon Video World Modelscited by 9
  18. 2026Do Joint Audio-Video Generation Models Understand Physics?cited by 8
  19. 2026CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narrativescited by 8
  20. 2026The Pulse of Motion: Measuring Physical Frame Rate from Visual Dynamicscited by 7
  21. 2026OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generationcited by 7
  22. 2026PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulationcited by 7
  23. 2026Parallel Decoding Distillation for Fast Image and Video Generationcited by 7
  24. 2026JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusioncited by 6
  25. 2026MentisOculi: Revealing the Limits of Reasoning with Mental Imagerycited by 6
  26. 2026DreamWorld: Unified World Modeling in Video Generationcited by 6
  27. 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Groundingcited by 6
  28. 2026Efficient Video Diffusion Models: Advancements and Challengescited by 6
  29. 2026Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generationcited by 6
  30. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 6
  31. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 6
  32. 2026Toward Native Multimodal Modeling: A Roadmapcited by 6
  33. 2026Native Audio-Visual Alignment for Generationcited by 6
  34. 2026Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generationcited by 6
  35. 2026Continuous Control of Editing Models via Adaptive-Origin Guidancecited by 5
  36. 2026AVControl: Efficient Framework for Training Audio-Visual Controlscited by 5
  37. 2026CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generationcited by 5
  38. 2026Improving Joint Audio-Video Generation with Cross-Modal Context Learningcited by 4

50 most cited of 213 All 213 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Text-to-Audio Generation Synchronized with Videos
  2. 20253MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation
  3. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generation
  4. 2026Diffusion Models for Joint Audio-Video Generation
  5. 2024STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment
  6. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation
  7. 2025UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generation
  8. 2025Training-Free Multimodal Guidance for Video to Audio Generation