mlsanityData to 14 Oct 2026

Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation

arXiv 2507.02271 ↗First posted 3 Jul 2025cs.CV cs.AI cs.MMcited by 4

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Video-to-Audio (V2A) Generation achieves significant progress and plays a crucial role in film and video post-production. However, current methods overlook the cinematic language, a critical component of artistic expression in filmmaking. As a result, their performance deteriorates in scenarios where Foley targets are only partially visible. To address this challenge, we propose a simple self-distillation approach to extend V2A models to cinematic language scenarios. By simulating the cinematic language variations, the student model learns to align the video features of training pairs with the same audio-visual correspondences, enabling it to effectively capture the associations between sounds and partial visual information. Our method not only achieves impressive improvements under partial visibility across all evaluation metrics, but also enhances performance on the large-scale V2A dataset, VGGSound.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 34 of its 40 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

34 of 40 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2016Densely Connected Convolutional Networkscited by 6,720
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  12. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
22 more
  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  2. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  3. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  4. 2021Taming Visually Guided Sound Generationcited by 136
  5. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  6. 2021Diverse Part Discovery: Occluded Person Re-identification with Part-Aware Transformercited by 120
  7. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  8. 2020Self-Supervised Scene De-occlusioncited by 96
  9. 2020Generating Visually Aligned Sound from Videoscited by 67
  10. 2020Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learningcited by 66
  11. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  12. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  13. 2024Temporally Aligned Audio for Video with Autoregressioncited by 52
  14. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  15. 2021Motion-aware Contrastive Video Representation Learning via Foreground-background Mergingcited by 45
  16. 2021Instance-wise Occlusion and Depth Orders in Natural Scenescited by 31
  17. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  18. 2022Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognitioncited by 27
  19. 2022Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foregroundcited by 27
  20. 2024STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignmentcited by 21
  21. 2024Don't Judge by the Look: Towards Motion Coherent Video Representationcited by 2
  22. 2024Human Orientation Estimation under Partial Observationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Hear What Matters! Text-conditioned Selective Video-to-Audio Generationcited by 3
  2. 2026FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scriptscited by 2
  3. 2026FoleyDesigner: Immersive Stereo Foley Generation with Precise Spatio-Temporal Alignment for Film Clipscited by 1
  4. 2026SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Semantic visually-guided acoustic highlighting with large vision-language models
  2. 2024Can video generation replace cinematographers? Research on the cinematic language of generated video
  3. 2025Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model
  4. 2023MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images
  5. 2025DreamFoley: Scalable VLMs for High-Fidelity Video-to-Audio Generation
  6. 2024SonicVisionLM: Playing Sound with Vision Language Models
  7. 2026LTX-2: Efficient Joint Audio-Visual Foundation Model
  8. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models