# AlignNet: A Unifying Approach to Audio-Visual Alignment

arXiv: https://arxiv.org/abs/2002.05070 · HTML: https://mlsanity.com/paper/2002.05070/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 13 of 13)

- [Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignment](https://mlsanity.com/paper/2307.12964/) (2023)
- [Long-Term Rhythmic Video Soundtracker](https://mlsanity.com/paper/2305.01319/) (2023)
- [TempCLR: Temporal Alignment Representation with Contrastive Learning](https://mlsanity.com/paper/2212.13738/) (2022)
- [PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scores](https://mlsanity.com/paper/2404.07336/) (2024)
- [TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation](https://mlsanity.com/paper/2110.13412/) (2021)
- [Let Your Video Listen to Your Music!](https://mlsanity.com/paper/2506.18881/) (2025)
- [MLink: Linking Black-Box Models from Multiple Domains for Collaborative Inference](https://mlsanity.com/paper/2209.13883/) (2022)
- [Audio-Visual Fusion Layers for Event Type Aware Video Recognition](https://mlsanity.com/paper/2202.05961/) (2022)
- [Video-to-Music Recommendation using Temporal Alignment of Segments](https://mlsanity.com/paper/2306.07187/) (2023)
- [ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localization](https://mlsanity.com/paper/2507.09945/) (2025)
- [Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models](https://mlsanity.com/paper/2607.09091/) (2026)
- [Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization](https://mlsanity.com/paper/2207.03190/) (2022)
- [Enhancing Explainability with Multimodal Context Representations for Smarter Robots](https://mlsanity.com/paper/2503.16467/) (2025)

## Abstract (arXiv)

We present AlignNet, a model that synchronizes videos with reference audios under non-uniform and irregular misalignments. AlignNet learns the end-to-end dense correspondence between each frame of a video and an audio. Our method is designed according to simple and well-established principles: attention, pyramidal processing, warping, and affinity function. Together with the model, we release a dancing dataset Dance50 for training and evaluation. Qualitative, quantitative and subjective evaluation results on dance-music alignment and speech-lip alignment demonstrate that our method far outperforms the state-of-the-art methods. Project video and code are available at https://jianrenw.github.io/AlignNet.