# Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation

arXiv: https://arxiv.org/abs/2507.02271 · HTML: https://mlsanity.com/paper/2507.02271/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 34 of its 40 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2306.17203/) (2023)
- [Diverse Part Discovery: Occluded Person Re-identification with Part-Aware Transformer](https://mlsanity.com/paper/2106.04095/) (2021)
- [FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds](https://mlsanity.com/paper/2407.01494/) (2024)
- [Self-Supervised Scene De-occlusion](https://mlsanity.com/paper/2004.02788/) (2020)
- [Generating Visually Aligned Sound from Videos](https://mlsanity.com/paper/2008.00820/) (2020)
- [Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning](https://mlsanity.com/paper/2009.05769/) (2020)
- [V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models](https://mlsanity.com/paper/2308.09300/) (2023)
- [Conditional Generation of Audio from Video via Foley Analogies](https://mlsanity.com/paper/2304.08490/) (2023)
- [Temporally Aligned Audio for Video with Autoregression](https://mlsanity.com/paper/2409.13689/) (2024)
- [Read, Watch and Scream! Sound Generation from Text and Video](https://mlsanity.com/paper/2407.05551/) (2024)
- [Motion-aware Contrastive Video Representation Learning via Foreground-background Merging](https://mlsanity.com/paper/2109.15130/) (2021)
- [Instance-wise Occlusion and Depth Orders in Natural Scenes](https://mlsanity.com/paper/2111.14562/) (2021)
- [Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity](https://mlsanity.com/paper/2407.10387/) (2024)
- [Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition](https://mlsanity.com/paper/2206.04790/) (2022)

## Built on by (most cited first, 4 of 4)

- [Hear What Matters! Text-conditioned Selective Video-to-Audio Generation](https://mlsanity.com/paper/2512.02650/) (2025)
- [FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts](https://mlsanity.com/paper/2603.19857/) (2026)
- [FoleyDesigner: Immersive Stereo Foley Generation with Precise Spatio-Temporal Alignment for Film Clips](https://mlsanity.com/paper/2604.05731/) (2026)
- [SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generation](https://mlsanity.com/paper/2601.15017/) (2026)

## Abstract (arXiv)

Video-to-Audio (V2A) Generation achieves significant progress and plays a crucial role in film and video post-production. However, current methods overlook the cinematic language, a critical component of artistic expression in filmmaking. As a result, their performance deteriorates in scenarios where Foley targets are only partially visible. To address this challenge, we propose a simple self-distillation approach to extend V2A models to cinematic language scenarios. By simulating the cinematic language variations, the student model learns to align the video features of training pairs with the same audio-visual correspondences, enabling it to effectively capture the associations between sounds and partial visual information. Our method not only achieves impressive improvements under partial visibility across all evaluation metrics, but also enhances performance on the large-scale V2A dataset, VGGSound.