mlsanityData to 7 Oct 2026

The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion

arXiv 2309.04509 ↗First posted 8 Sep 2023cs.SD cs.CV cs.GRcited by 41

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In recent years, video generation has become a prominent generative tool and has drawn significant attention. However, there is little consideration in audio-to-video generation, though audio contains unique qualities like temporal semantics and magnitude. Hence, we propose The Power of Sound (TPoS) model to incorporate audio input that includes both changeable temporal semantics and magnitude. To generate video frames, TPoS utilizes a latent stable diffusion model with textual semantic information, which is then guided by the sequential audio embedding from our pretrained Audio Encoder. As a result, this method produces audio reactive video contents. We demonstrate the effectiveness of TPoS across various tasks and compare its results with current state-of-the-art techniques in the field of audio-to-video generation. More examples are available at https://ku-vai.github.io/TPoS/

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 34 of its 39 references that are in the map. First posted on arXiv 8 Sep 2023.

Builds on

34 of 39 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2016Improved Techniques for Training GANscited by 2,938
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2017The Kinetics Human Action Video Datasetcited by 1,894
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
22 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  3. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  4. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  5. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  6. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  7. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  8. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  9. 2022Blended Latent Diffusioncited by 430
  10. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  11. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  12. 2021Audio-Driven Emotional Video Portraitscited by 180
  13. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  14. 2022Uncovering the Disentanglement Capability in Text-to-Image Diffusion Modelscited by 116
  15. 2021Aligning Latent and Image Spaces to Connect the Unconnectablecited by 70
  16. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  17. 2021Sound-Guided Semantic Image Manipulationcited by 35
  18. 2022Sound-Guided Semantic Video Generationcited by 35
  19. 2020Sound2Sight: Generating Visual Dynamics from Sound and Contextcited by 28
  20. 2022The Stable Artist: Steering Semantics in Diffusion Latent Spacecited by 23
  21. 2021Tr"aumerAI: Dreaming Music with StyleGANcited by 11
  22. 2019Automatic Realistic Music Video Generation from Segments of Youtube Videoscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 15; 2025: 14; 2026: 6.

41 papers.

Sort:

By area:

  1. 2023A Survey on Video Diffusion Modelscited by 203
  2. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  3. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  4. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  5. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  6. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  7. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  8. 2023ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separationcited by 37
  9. 2024Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generationcited by 35
  10. 2024Audio-Synchronized Visual Animationcited by 32
  11. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  12. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generationcited by 23
29 more
  1. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  2. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  3. 2025UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generationcited by 21
  4. 2024Language-Guided Joint Audio-Visual Editing via One-Shot Adaptationcited by 18
  5. 2023Brain-Conditional Multimodal Synthesis: A Survey and Taxonomycited by 15
  6. 2024SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Modelscited by 15
  7. 2025Audio-Sync Video Generation with Multi-Stream Temporal Controlcited by 14
  8. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 13
  9. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 12
  10. 2023Diffusion for Natural Image Mattingcited by 11
  11. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 11
  12. 2023Synthetic Shifts to Initial Seed Vector Exposes the Brittle Nature of Latent-Based Diffusion Modelscited by 9
  13. 2025Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interactioncited by 9
  14. 2023CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modelingcited by 7
  15. 2025Diffusion Classifiers Understand Compositionality, but Conditions Applycited by 7
  16. 2025KeyVID: Keyframe-Aware Video Diffusion for Audio-Synchronized Visual Animationcited by 6
  17. 2025Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformerscited by 6
  18. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 6
  19. 2026Inference-Time Scaling for Joint Audio-Video Generationcited by 3
  20. 2024Sequential Contrastive Audio-Visual Learningcited by 2
  21. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  22. 2025Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigmcited by 2
  23. 2025ProAV-DiT: A Projected Latent Diffusion Transformer for Efficient Synchronized Audio-Video Generationcited by 2
  24. 2025Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removalcited by 2
  25. 2024VCoME: Verbal Video Composition with Multimodal Editing Effectscited by 1
  26. 2025KDC-Diff: A Latent-Aware Diffusion Model with Knowledge Retention for Memory-Efficient Image Generationcited by 0
  27. 2025Seeing What You Say: Expressive Image Generation from Speechcited by 0
  28. 2026Bring Music The Horizon: Music-Driven 360^circ Video Generationcited by 0
  29. 2026TurboT2VA: Fast Large-Scale Text-to-Video-Audio Generation via Score-Regularized Consistency Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffusion Models for Joint Audio-Video Generation
  2. 20253MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation
  3. 2023AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion
  4. 2024LoVA: Long-form Video-to-Audio Generation
  5. 2026HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion
  6. 2026Soundwich: Video Generation with Layered and Controllable Audio
  7. 2024Video-to-Audio Generation with Fine-grained Temporal Semantics
  8. 2024A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation