# Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation

arXiv: https://arxiv.org/abs/2412.08161 · HTML: https://mlsanity.com/paper/2412.08161/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 35 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 6 of 6)

- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation](https://mlsanity.com/paper/2307.13236/) (2023)
- [Contrastive Conditional Latent Diffusion for Audio-visual Segmentation](https://mlsanity.com/paper/2307.16579/) (2023)
- [Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation](https://mlsanity.com/paper/2309.09501/) (2023)
- [Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation](https://mlsanity.com/paper/2312.06462/) (2023)

## Built on by (most cited first, 1 of 1)

- [Zero-shot Compositional Action Recognition with Neural Logic Constraints](https://mlsanity.com/paper/2508.02320/) (2025)

## Abstract (arXiv)

Audio-visual video segmentation (AVVS) aims to generate pixel-level maps of sound-producing objects that accurately align with the corresponding audio. However, existing methods often face temporal misalignment, where audio cues and segmentation results are not temporally coordinated. Audio provides two critical pieces of information: i) target object-level details and ii) the timing of when objects start and stop producing sounds. Current methods focus more on object-level information but neglect the boundaries of audio semantic changes, leading to temporal misalignment. To address this issue, we propose a Collaborative Hybrid Propagator Framework~(Co-Prop). This framework includes two main steps: Preliminary Audio Boundary Anchoring and Frame-by-Frame Audio-Insert Propagation. To Anchor the audio boundary, we employ retrieval-assist prompts with Qwen large language models to identify control points of audio semantic changes. These control points split the audio into semantically consistent audio portions. After obtaining the control point lists, we propose the Audio Insertion Propagator to process each audio portion using a frame-by-frame audio insertion propagation and matching approach. We curated a compact dataset comprising diverse source conversion cases and devised a metric to assess alignment rates. Compared to traditional simultaneous processing methods, our approach reduces memory requirements and facilitates frame alignment. Experimental results demonstrate the effectiveness of our approach across three datasets and two backbones. Furthermore, our method can be integrated with existing AVVS approaches, offering plug-and-play functionality to enhance their performance.