# PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling

arXiv: https://arxiv.org/abs/2505.23155 · HTML: https://mlsanity.com/paper/2505.23155/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 74 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://mlsanity.com/paper/2406.11816/) (2024)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?](https://mlsanity.com/paper/2501.05510/) (2025)
- [ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities](https://mlsanity.com/paper/2305.11172/) (2023)
- [VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision](https://mlsanity.com/paper/2412.14446/) (2024)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [Online Action Detection](https://mlsanity.com/paper/1604.06506/) (2016)
- [Real-time Online Video Detection with Temporal Smoothing Transformers](https://mlsanity.com/paper/2209.09236/) (2022)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos](https://mlsanity.com/paper/2411.19772/) (2024)
- [Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline](https://mlsanity.com/paper/2303.12930/) (2023)
- [Egocentric Audio-Visual Object Localization](https://mlsanity.com/paper/2303.13471/) (2023)
- [MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing](https://mlsanity.com/paper/2111.12374/) (2021)
- [Memory-and-Anticipation Transformer for Online Action Understanding](https://mlsanity.com/paper/2308.07893/) (2023)
- [360+x: A Panoptic Multi-modal Scene Understanding Dataset](https://mlsanity.com/paper/2404.00989/) (2024)
- [Audio-Visual Class-Incremental Learning](https://mlsanity.com/paper/2308.11073/) (2023)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization](https://mlsanity.com/paper/2210.05060/) (2022)
- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)

## Built on by (most cited first, 4 of 4)

- [Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation](https://mlsanity.com/paper/2508.04418/) (2025)
- [Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation](https://mlsanity.com/paper/2602.03892/) (2026)
- [Event ActivityNet: A Large-Scale Simulated-Event Benchmark for Untrimmed Action Understanding](https://mlsanity.com/paper/2608.01948/) (2026)
- [SCoPE: Training-Free Audio-Visual Event Perception via Sparse Cross-Modal Prior Exchange](https://mlsanity.com/paper/2608.07923/) (2026)

## Abstract (arXiv)

Audio-visual event parsing plays a crucial role in understanding multimodal video content, but existing methods typically rely on offline processing of entire videos with huge model sizes, limiting their real-time applicability. We introduce Online Audio-Visual Event Parsing (On-AVEP), a novel paradigm for parsing audio, visual, and audio-visual events by sequentially analyzing incoming video streams. The On-AVEP task necessitates models with two key capabilities: (1) Accurate online inference, to effectively distinguish events with unclear and limited context in online settings, and (2) Real-time efficiency, to balance high performance with computational constraints. To cultivate these, we propose the Predictive Future Modeling (PreFM) framework featured by (a) predictive multimodal future modeling to infer and integrate beneficial future audio-visual cues, thereby enhancing contextual understanding and (b) modality-agnostic robust representation along with focal temporal prioritization to improve precision and generalization. Extensive experiments on the UnAV-100 and LLP datasets show PreFM significantly outperforms state-of-the-art methods by a large margin with significantly fewer parameters, offering an insightful approach for real-time multimodal video understanding. Code is available at https://github.com/XiaoYu-1123/PreFM.