mlsanityData to 14 Oct 2026

Out of Sight, Still in Mind: Token Compression for Omni-LLMs

arXiv 2607.21179 ↗First posted 23 Jul 2026cs.CVcited by 2

On the map: Efficient LLM inference and serving → Audio and omni-modal token compression

Abstract (arXiv)

The goal of this paper is to reduce the input token cost of Omni-modal large language models (Omni-LLMs) at inference time. Omni-LLMs reason jointly over audio, video and text, but the cost of the three streams is highly unbalanced: visual tokens account for the vast majority of the input, and are highly redundant. In this paper, we propose ReMo, a training-free framework that compresses visual tokens by redistributing their information across modalities: a visual token is kept only if its information appears nowhere else. ReMo achieves this in two ways: (i) it aligns audio and video in a common embedding space, and removes visual tokens already explained by the audio or by other visual tokens; and (ii) it replaces object-level visual tokens with compact text proxies, short descriptions of each object and its location, conveying the same content in far fewer tokens. On Qwen2.5-Omni at two model scales, ReMo removes 54% of the input tokens with no loss in accuracy. Indeed, it slightly exceeds the full-token model, reaching 101.2% and 101.3% of its average accuracy over five audio-visual benchmarks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 39 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

27 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance14 May
  2. 2026Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs12 May
  3. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models4 Feb

Other references · 24

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  8. 2022Token Merging: Your ViT But Fastercited by 1,021
  9. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  11. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  12. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
12 more
  1. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  2. 2024What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detectorcited by 45
  3. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 30
  4. 2026Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencecited by 30
  5. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 10
  6. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6
  7. 2026Closing the Modality Gap Aligns Group-Wise Semanticscited by 5
  8. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMscited by 5
  9. 2025Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decodingcited by 4
  10. 2026ViKey: Enhancing Temporal Understanding in Videos via Visual Promptingcited by 3
  11. 2026TextME: Bridging Unseen Modalities Through Text Descriptionscited by 2
  12. 2026LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergencecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026OmniKVQuant: KV Cache Quantization for Omni-LLMscited by 0
  2. 2026Learning to Reason with Compressed Context: Ground-Truth-Free Adaptation of OmniLLMs via Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models
  2. 2026OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
  3. 2026OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
  4. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models
  5. 2026OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models
  6. 2026OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models
  7. 2026OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs
  8. 2025A Survey of Token Compression for Efficient Multimodal Large Language Models