# LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

arXiv: https://arxiv.org/abs/2608.26820 · HTML: https://mlsanity.com/paper/2608.26820/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Progressive Prompts: Continual Learning for Language Models](https://mlsanity.com/paper/2301.12314/) (2023)
- [Task-Specific Skill Localization in Fine-tuned Language Models](https://mlsanity.com/paper/2302.06600/) (2023)
- [Compressing Visual-linguistic Model via Knowledge Distillation](https://mlsanity.com/paper/2104.02096/) (2021)
- [EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning](https://mlsanity.com/paper/2210.07795/) (2022)
- [Multimodal Instruction Tuning with Conditional Mixture of LoRA](https://mlsanity.com/paper/2402.15896/) (2024)
- [Improved Feature Distillation via Projector Ensemble](https://mlsanity.com/paper/2210.15274/) (2022)
- [Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement](https://mlsanity.com/paper/2412.01282/) (2024)
- [Information Theoretic Representation Distillation](https://mlsanity.com/paper/2112.00459/) (2021)
- [SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture](https://mlsanity.com/paper/2410.07739/) (2024)
- [AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability](https://mlsanity.com/paper/2405.14129/) (2024)
- [VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition](https://mlsanity.com/paper/2408.16930/) (2024)
- [InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer](https://mlsanity.com/paper/2511.15967/) (2025)

## Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) exhibit strong generalization, visual instruction tuning for downstream tasks inevitably causes catastrophic forgetting, impairing overall generalization. While existing methods regulate weight updates to reduce forgetting, they overlook the fundamental cross-modal alignment in MLLMs. Based on prior work and our observations, we argue that cross-modal alignment is implicitly captured in the information-compression trajectory. To preserve the alignment flow embedded in the trajectory, we propose LLaVAFlow, an information-theoretic distillation framework. First, we compress the mutual information between the extracted relations and MLLM embeddings, encouraging a learnable module to produce a refined alignment flow that benefits downstream tasks. Second, we maximize the mutual information between the extracted alignment flows of the pretrained and fine-tuned MLLMs, enabling the transfer of compact alignment information. Extensive experiments show that LLaVAFlow is an effective plug-and-play framework that preserves alignment flow and enhances both downstream performance and generalization.