# Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2603.21426 · HTML: https://mlsanity.com/paper/2603.21426/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 58 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 9 of 9)

- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [MobileVLM V2: Faster and Stronger Baseline for Vision Language Model](https://mlsanity.com/paper/2402.03766/) (2024)
- [Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation](https://mlsanity.com/paper/2105.08919/) (2021)
- [MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices](https://mlsanity.com/paper/2312.16886/) (2023)
- [LLAVADI: What Matters For Multimodal Large Language Models Distillation](https://mlsanity.com/paper/2407.19409/) (2024)
- [LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation](https://mlsanity.com/paper/2502.00896/) (2025)
- [Prompt Tuning based Adapter for Vision-Language Model Adaption](https://mlsanity.com/paper/2303.15234/) (2023)
- [Cosine Similarity Knowledge Distillation for Individual Class Information Transfer](https://mlsanity.com/paper/2311.14307/) (2023)

## Built on by (most cited first, 1 of 1)

- [Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight](https://mlsanity.com/paper/2606.00424/) (2026)

## Abstract (arXiv)

Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is challenging, as some samples may be noisy while others are subject to teacher uncertainty. This motivates the need for adaptively balancing data and teacher supervision. We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that adaptively modulates how much the student relies on teacher guidance. Specifically, we formulate teacher--student learning from a unified Bayesian perspective and interpret teacher supervision as a Gibbs prior over student activations. This yields a closed-form, uncertainty-aware weighting mechanism and supports arbitrary distillation objectives and their combinations. Extensive experiments on multimodal VQA benchmarks demonstrate that distilling student Vision-Language Models from a large teacher VLM consistently improves performance. The results show that Beta-KD outperforms existing knowledge distillation methods. The code is available at https://github.com/Jingchensun/beta-kd.