# Out of Sight, Still in Mind: Token Compression for Omni-LLMs

arXiv: https://arxiv.org/abs/2607.21179 · HTML: https://mlsanity.com/paper/2607.21179/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Placed in

- Efficient LLM inference and serving / [Audio and omni-modal token compression](https://mlsanity.com/inference/audio-and-omni-modal-token-compression/)

## Builds on (most cited first, 27 of 27)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [VITA: Towards Open-Source Interactive Omni Multimodal LLM](https://mlsanity.com/paper/2408.05211/) (2024)
- [WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs](https://mlsanity.com/paper/2502.04326/) (2025)
- [Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities](https://mlsanity.com/paper/2505.17862/) (2025)
- [What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector](https://mlsanity.com/paper/2408.15857/) (2024)
- [AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding](https://mlsanity.com/paper/2505.20862/) (2025)
- [Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence](https://mlsanity.com/paper/2604.24954/) (2026)
- [OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models](https://mlsanity.com/paper/2602.04804/) (2026)
- [Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings](https://mlsanity.com/paper/2602.06218/) (2026)
- [On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs](https://mlsanity.com/paper/2603.14337/) (2026)
- [Closing the Modality Gap Aligns Group-Wise Semantics](https://mlsanity.com/paper/2601.18525/) (2026)
- [Probing Cross-modal Information Hubs in Audio-Visual LLMs](https://mlsanity.com/paper/2605.10815/) (2026)
- [Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding](https://mlsanity.com/paper/2506.09522/) (2025)
- [OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance](https://mlsanity.com/paper/2605.14458/) (2026)
- [ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting](https://mlsanity.com/paper/2603.23186/) (2026)
- [Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs](https://mlsanity.com/paper/2605.11605/) (2026)
- [TextME: Bridging Unseen Modalities Through Text Descriptions](https://mlsanity.com/paper/2602.03098/) (2026)
- [LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2601.04658/) (2026)

## Built on by (most cited first, 2 of 2)

- [OmniKVQuant: KV Cache Quantization for Omni-LLMs](https://mlsanity.com/paper/2609.11582/) (2026)
- [Learning to Reason with Compressed Context: Ground-Truth-Free Adaptation of OmniLLMs via Self-Distillation](https://mlsanity.com/paper/2609.39953/) (2026)

## Abstract (arXiv)

The goal of this paper is to reduce the input token cost of Omni-modal large language models (Omni-LLMs) at inference time. Omni-LLMs reason jointly over audio, video and text, but the cost of the three streams is highly unbalanced: visual tokens account for the vast majority of the input, and are highly redundant. In this paper, we propose ReMo, a training-free framework that compresses visual tokens by redistributing their information across modalities: a visual token is kept only if its information appears nowhere else. ReMo achieves this in two ways: (i) it aligns audio and video in a common embedding space, and removes visual tokens already explained by the audio or by other visual tokens; and (ii) it replaces object-level visual tokens with compact text proxies, short descriptions of each object and its location, conveying the same content in far fewer tokens. On Qwen2.5-Omni at two model scales, ReMo removes 54% of the input tokens with no loss in accuracy. Indeed, it slightly exceeds the full-token model, reaching 101.2% and 101.3% of its average accuracy over five audio-visual benchmarks.