# OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs

arXiv: https://arxiv.org/abs/2606.07577 · HTML: https://mlsanity.com/paper/2606.07577/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 26 May 2026.

## Builds on (most cited first, 19 of 19)

- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [LongVILA: Scaling Long-Context Visual Language Models for Long Videos](https://mlsanity.com/paper/2408.10188/) (2024)
- [Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding](https://mlsanity.com/paper/2409.14485/) (2024)
- [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024)
- [Streaming Video Question-Answering with In-context Video KV-Cache Retrieval](https://mlsanity.com/paper/2503.00540/) (2025)
- [StreamForest: Efficient Online Video Understanding with Persistent Event Memory](https://mlsanity.com/paper/2509.24871/) (2025)
- [InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding](https://mlsanity.com/paper/2506.15745/) (2025)
- [StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding](https://mlsanity.com/paper/2508.15717/) (2025)
- [Squeezed Attention: Accelerating Long Context Length LLM Inference](https://mlsanity.com/paper/2411.09688/) (2024)
- [Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understanding](https://mlsanity.com/paper/2503.18478/) (2025)
- [StreamingTOM: Streaming Token Compression for Efficient Video Understanding](https://mlsanity.com/paper/2510.18269/) (2025)
- [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2601.14724/) (2026)
- [StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression](https://mlsanity.com/paper/2511.07278/) (2025)
- [LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs](https://mlsanity.com/paper/2603.19217/) (2026)
- [LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation](https://mlsanity.com/paper/2509.09754/) (2025)

## Built on by (most cited first, 4 of 4)

- [SPACE: Sparse Predictive Attractor via Counterfactual Eviction for Streaming Video Memory](https://mlsanity.com/paper/2609.32592/) (2026)
- [OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models](https://mlsanity.com/paper/2609.37052/) (2026)
- [AS2D: Accelerating On-Demand Audio Understanding on Mobile Devices](https://mlsanity.com/paper/2609.37617/) (2026)
- [LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception](https://mlsanity.com/paper/2609.39938/) (2026)

## Abstract (arXiv)

Audio-visual large language models (LLMs) hold strong promise for long-form video understanding, yet their long-video inference is fundamentally limited by the linear growth of video tokens and key-value (KV) caches. We present OmniMem, a memory-efficient streaming framework designed specifically for audio-visual LLMs. Unlike existing compression methods that treat all tokens uniformly, OmniMem introduces a modality-aware memory allocation strategy that separately manages visual and audio contexts, addressing the severe token imbalance between the two modalities. OmniMem further preserves informative and non-redundant KV states through perturbation-aware memory selection, enabling compact memory without sacrificing long-range understanding. To strengthen compression under realistic deployment constraints, we also explore budget-aware fine-tuning, which encourages the model to consolidate useful information into retained memory. Experiments on VideoMME Long, LVBench, and LVOmniBench with video-SALMONN 2+ and Qwen-2.5-Omni show that OmniMem consistently improves over strong training-free compression baselines by 2-4% absolute accuracy under the same memory budgets, with an additional 1-2% gain after fine-tuning.