mlsanityData to 14 Oct 2026

Eddy-VL 1.9B: Structural Pruning and Layered Distillation for Edge-Deployable Multimodal Embedding

arXiv 2607.16316 ↗First posted 15 Jul 2026cs.CVcited by 0

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

In this report, we introduce Eddy-VL 1.9B, a compressed multimodal embedding model built on Qwen3-VL-Embedding-2B for offline, edge-deployable vision-language retrieval. Eddy-VL targets air-gapped forensic and investigative settings where cloud APIs are unavailable and low latency is essential. Compression combines (i) probe-driven structural pruning that removes four redundant text-decoder layers (28 to 24) ranked by adjacent-layer linear CKA, and (ii) layered knowledge distillation with hole-covering teacher-student mappings, mid-layer attention-map 1-CKA, and final-layer MSE and cosine losses with Matryoshka dimensions {128, 256, 512, 1024, 2048}. The released model contains 1,926,188,032 parameters (3.85 GB bf16), representing approximately 9.5% fewer parameters than the 2.13B teacher model. Empirical evaluations on MMEB-V2 (78 tasks, VLM2Vec protocol) show that Eddy-VL achieves an overall score of 63.2 compared with 68.9 for the teacher, retaining 91.7% of the teacher's performance while recovering 6.4 of the 12.1 points lost through pruning alone (56.8). Compositional reasoning performance remains close to the teacher on SugarCrepe (86.1 vs. 86.4), MR2-Bench (24.5 vs. 24.7), and ARO (59.5 vs. 60.4), while Winoground group performance (6.8 vs. 8.5) remains the primary limitation. Depth pruning also reduces forward latency by approximately 10% (150.0 to 136.4 ms per image on NVIDIA DGX Spark using FlashAttention-2). We present the architecture, compression methodology, training procedures, and evaluation results, demonstrating the effectiveness of Eddy-VL for multimodal retrieval under constrained edge deployment. Model weights and inference code are publicly available on Hugging Face.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 16 of its 20 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

16 of 20 references.

References · 16

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  5. 2019Relational Knowledge Distillationcited by 889
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  7. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  8. 2022Matryoshka Representation Learningcited by 282
  9. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  10. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  11. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  12. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
4 more
  1. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  2. 2024Rethinking Centered Kernel Alignment in Knowledge Distillationcited by 17
  3. 2025With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide Youcited by 16
  4. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient VLMs: Information-Theoretic Driven Compression via Adaptive Structural Pruning
  2. 2026Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression
  3. 2026Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency
  4. 2026Hierarchical Compression of Vision-Language Model Benchmarks
  5. 2025ABC: Achieving Better Control of Multimodal Embeddings using VLMs
  6. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models
  7. 2026LensVLM: Selective Context Expansion for Compressed Visual Representation of Text
  8. 2026Bifocal: Training-Free Mixed Granularity Visual Token Pruning for Multimodal LLMs