# Tri-Ergon: Fine-grained Video-to-Audio Generation with Multi-modal Conditions and LUFS Control

arXiv: https://arxiv.org/abs/2412.20378 · HTML: https://mlsanity.com/paper/2412.20378/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 36 references that are in the map.
- First posted on arXiv 29 Dec 2024.

## Builds on (most cited first, 11 of 11)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Noise2Music: Text-conditioned Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.03917/) (2023)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Progressive Distillation for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2202.00512/) (2022)
- [Contrastive Conditional Latent Diffusion for Audio-visual Segmentation](https://mlsanity.com/paper/2307.16579/) (2023)
- [Controllable Music Production with Diffusion Models and Guidance Gradients](https://mlsanity.com/paper/2311.00613/) (2023)
- [You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet](https://mlsanity.com/paper/2405.21022/) (2024)

## Built on by (most cited first, 8 of 8)

- [DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos](https://mlsanity.com/paper/2503.22208/) (2025)
- [StoryBlender: Inter-Shot Consistent and Editable 3D Storyboard with Spatial-temporal Dynamics](https://mlsanity.com/paper/2604.03315/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [OmniSonic: Towards Universal and Holistic Audio Generation from Video and Text](https://mlsanity.com/paper/2604.04348/) (2026)
- [Enhance Generation Quality of Flow Matching V2A Model via Multi-Step CoT-Like Guidance and Combined Preference Optimization](https://mlsanity.com/paper/2503.22200/) (2025)
- [Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks](https://mlsanity.com/paper/2505.20038/) (2025)
- [LD-LAudio-V1: Video-to-Long-Form-Audio Generation Extension with Dual Lightweight Adapters](https://mlsanity.com/paper/2508.11074/) (2025)
- [HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion](https://mlsanity.com/paper/2608.11913/) (2026)

## Abstract (arXiv)

Video-to-audio (V2A) generation utilizes visual-only video features to produce realistic sounds that correspond to the scene. However, current V2A models often lack fine-grained control over the generated audio, especially in terms of loudness variation and the incorporation of multi-modal conditions. To overcome these limitations, we introduce Tri-Ergon, a diffusion-based V2A model that incorporates textual, auditory, and pixel-level visual prompts to enable detailed and semantically rich audio synthesis. Additionally, we introduce Loudness Units relative to Full Scale (LUFS) embedding, which allows for precise manual control of the loudness changes over time for individual audio channels, enabling our model to effectively address the intricate correlation of video and audio in real-world Foley workflows. Tri-Ergon is capable of creating 44.1 kHz high-fidelity stereo audio clips of varying lengths up to 60 seconds, which significantly outperforms existing state-of-the-art V2A methods that typically generate mono audio for a fixed duration.