mlsanityData to 7 Oct 2026

Towards Long-window Anchoring in Vision-Language Model Distillation

arXiv 2512.21576 ↗First posted 25 Dec 2025cs.CV cs.AI cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

While large vision-language models (VLMs) demonstrate strong long-context understanding, their prevalent small branches fail on linguistics-photography alignment for a limited window size. We discover that knowledge distillation improves students' capability as a complement to Rotary Position Embeddings (RoPE) on window sizes (anchored from large models). Building on this insight, we propose LAid, which directly aims at the transfer of long-range attention mechanisms through two complementary components: (1) a progressive distance-weighted attention matching that dynamically emphasizes longer position differences during training, and (2) a learnable RoPE response gain modulation that selectively amplifies position sensitivity where needed. Extensive experiments across multiple model families demonstrate that LAid-distilled models achieve up to 3.2 times longer effective context windows compared to baseline small models, while maintaining or improving performance on standard VL benchmarks. Spectral analysis also suggests that LAid successfully preserves crucial low-frequency attention components that conventional methods fail to transfer. Our work not only provides practical techniques for building more efficient long-context VLMs but also offers theoretical insights into how positional understanding emerges and transfers during distillation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 28 references that are in the map. First posted on arXiv 25 Dec 2025.

Builds on

23 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillation28 Jul

Other references · 21

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2024Yi: Open Foundation Models by 01.AIcited by 706
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  7. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  8. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  9. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  10. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  11. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  12. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
9 more
  1. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  2. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  3. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  4. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  5. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  6. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  7. 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14
  8. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13
  9. 2024VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognitioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Can Vision Replace Text in Working Memory? Evidence from Spatial n-Back in Vision-Language Modelscited by 0
  2. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models
  2. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding
  3. 2026Large Vision-Language Models Get Lost in Attention
  4. 2026Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs
  5. 2025Attention Guided Alignment in Efficient Vision-Language Models
  6. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models
  7. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  8. 2024Visual Context Window Extension: A New Perspective for Long Video Understanding