# Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation

arXiv: https://arxiv.org/abs/2507.04959 · HTML: https://mlsanity.com/paper/2507.04959/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion](https://mlsanity.com/paper/2403.06976/) (2024)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [InstanceDiffusion: Instance-level Control for Image Generation](https://mlsanity.com/paper/2402.03290/) (2024)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds](https://mlsanity.com/paper/2407.01494/) (2024)
- [I Hear Your True Colors: Image Guided Audio Generation](https://mlsanity.com/paper/2211.03089/) (2022)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [MixGen: A New Multi-Modal Data Augmentation](https://mlsanity.com/paper/2206.08358/) (2022)
- [Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts](https://mlsanity.com/paper/2403.08268/) (2024)
- [Learning Multimodal Data Augmentation in Feature Space](https://mlsanity.com/paper/2212.14453/) (2022)
- [Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model](https://mlsanity.com/paper/2404.16305/) (2024)

## Built on by (most cited first, 2 of 2)

- [Hear What Matters! Text-conditioned Selective Video-to-Audio Generation](https://mlsanity.com/paper/2512.02650/) (2025)
- [Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy](https://mlsanity.com/paper/2609.34381/) (2026)

## Abstract (arXiv)

Video-to-audio (V2A) generation shows great potential in fields such as film production. Despite significant advances, current V2A methods relying on global video information struggle with complex scenes and generating audio tailored to specific objects. To address these limitations, we introduce Hear-Your-Click, an interactive V2A framework enabling users to generate sounds for specific objects by clicking on the frame. To achieve this, we propose Object-aware Contrastive Audio-Visual Fine-tuning (OCAV) with a Mask-guided Visual Encoder (MVE) to obtain object-level visual features aligned with audio. Furthermore, we tailor two data augmentation strategies, Random Video Stitching (RVS) and Mask-guided Loudness Modulation (MLM), to enhance the model's sensitivity to segmented objects. To measure audio-visual correspondence, we designed a new evaluation metric, the CAV score. Extensive experiments demonstrate that our framework offers more precise control and improves generation performance across various metrics. Project Page: https://github.com/SynapGrid/Hear-Your-Click