# Learning to Reason with Compressed Context: Ground-Truth-Free Adaptation of OmniLLMs via Self-Distillation

arXiv: https://arxiv.org/abs/2609.39953 · HTML: https://mlsanity.com/paper/2609.39953/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)
- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 30 of 32)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [VITA: Towards Open-Source Interactive Omni Multimodal LLM](https://mlsanity.com/paper/2408.05211/) (2024)
- [Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities](https://mlsanity.com/paper/2505.17862/) (2025)
- [Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation](https://mlsanity.com/paper/2605.18740/) (2026)
- [HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context](https://mlsanity.com/paper/2506.21277/) (2025)
- [Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillation](https://mlsanity.com/paper/2606.08719/) (2026)
- [OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains](https://mlsanity.com/paper/2606.14702/) (2026)
- [DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression](https://mlsanity.com/paper/2603.15685/) (2026)
- [OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing](https://mlsanity.com/paper/2606.15920/) (2026)
- [RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models](https://mlsanity.com/paper/2607.24447/) (2026)
- [Self-Supervised Visual On-Policy Distillation](https://mlsanity.com/paper/2608.14144/) (2026)
- [Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding](https://mlsanity.com/paper/2608.25356/) (2026)
- [OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance](https://mlsanity.com/paper/2605.14458/) (2026)
- [OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models](https://mlsanity.com/paper/2605.18041/) (2026)
- [Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation](https://mlsanity.com/paper/2607.23125/) (2026)
- [Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs](https://mlsanity.com/paper/2605.11605/) (2026)
- [OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models](https://mlsanity.com/paper/2605.12056/) (2026)
- [Stage-adaptive Token Selection for Efficient Omni-modal LLMs](https://mlsanity.com/paper/2605.20035/) (2026)
- [AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression](https://mlsanity.com/paper/2606.24286/) (2026)
- [OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models](https://mlsanity.com/paper/2608.03812/) (2026)
- [O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding](https://mlsanity.com/paper/2605.26584/) (2026)
- [OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models](https://mlsanity.com/paper/2607.03050/) (2026)
- [Out of Sight, Still in Mind: Token Compression for Omni-LLMs](https://mlsanity.com/paper/2607.21179/) (2026)
- [Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression](https://mlsanity.com/paper/2608.02109/) (2026)
- [Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs](https://mlsanity.com/paper/2608.08794/) (2026)
- [OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models](https://mlsanity.com/paper/2607.23193/) (2026)
- [Omni-Prune: Query-Aware Unified Token Pruning for Efficient Omnimodal Large Language Models](https://mlsanity.com/paper/2607.23445/) (2026)
- [OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs](https://mlsanity.com/paper/2607.25669/) (2026)

## Abstract (arXiv)

Omni-modal large language models (OmniLLMs) enable unified audio-video understanding, but their long multimodal token sequences make deployment computationally expensive. Token compression reduces this cost, yet aggressive compression often lowers accuracy. Existing works predominantly focus on designing better compression mechanisms; however, adapting the underlying language model to reason effectively over the remaining compressed context remains under-explored. To address this, we propose CAFD (Compressed-Context Adaptation via Full-Context Distillation), a ground-truth-free self-distillation framework that adapts OmniLLMs to fixed compression pipelines without requiring reference answers, rationales, or correctness rewards. CAFD leverages the full-token view of the same multimodal sample as a source of privileged information: a full-context self-teacher provides soft target supervision to a compressed-context student along the student's on-policy trajectory. Evaluated on Qwen2.5-Omni-7B across five audio-video benchmarks, five compression pipelines, and five deployment budgets, CAFD demonstrates consistent gains, improving 120 out of 125 conditions with an average accuracy boost of 1.44 points and recovering 26.9% of the accuracy gap on average. These results demonstrate that the proposed ground-truth-free adaptation offers an effective and practical route to improving the accuracy-efficiency trade-off in deployed OmniLLMs.