mlsanityData to 14 Oct 2026

Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting

arXiv 2505.00029 ↗First posted 27 Apr 2025cs.CL cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Vision Language Models have demonstrated impressive versatile capabilities through extensive multimodal pre-training, but face significant limitations when incorporating specialized knowledge domains beyond their training distribution. These models struggle with a fundamental dilemma: direct adaptation approaches that inject domain-specific knowledge often trigger catastrophic forgetting of foundational visual-linguistic abilities. We introduce Structured Dialogue Fine-Tuning (SDFT), an effective approach that effectively injects domain-specific knowledge while minimizing catastrophic forgetting. Drawing inspiration from supervised fine-tuning in LLMs and subject-driven personalization in text-to-image diffusion models, our method employs a three-phase dialogue structure: Foundation Preservation reinforces pre-trained visual-linguistic alignment through caption tasks; Contrastive Disambiguation introduces carefully designed counterfactual examples to maintain semantic boundaries; and Knowledge Specialization embeds specialized information through chain-of-thought reasoning. Experimental results across multiple domains confirm SDFT's effectiveness in balancing specialized knowledge acquisition with general capability retention. Our key contributions include a data-centric dialogue template that balances foundational alignment with targeted knowledge integration, a weighted multi-turn supervision framework, and comprehensive evaluation across diverse knowledge types.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 45 references that are in the map. First posted on arXiv 27 Apr 2025.

Builds on

31 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models17 Oct
  2. 2024Yo'LLaVA: Your Personalized Language and Vision Assistant13 Jun
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  10. 2023BloombergGPT: A Large Language Model for Financecited by 939
  11. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  12. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
16 more
  1. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  2. 2023Fine-tuning Language Models for Factualitycited by 221
  3. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  4. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  5. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  6. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 117
  7. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 88
  8. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  9. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 64
  10. 2023Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Modelscited by 58
  11. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  12. 2024From Skepticism to Acceptance: Simulating the Attitude Dynamics Toward Fake Newscited by 51
  13. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 51
  14. 2024Imagine yourself: Tuning-Free Personalized Image Generationcited by 30
  15. 2023KILM: Knowledge Injection into Encoder-Decoder Language Modelscited by 19
  16. 2025InsightVision: A Comprehensive, Multi-Level Chinese-based Benchmark for Evaluating Implicit Visual Semantics in Large Vision Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MERGETUNE: Continued Fine-Tuning of Vision-Language Models
  2. 2025Infusing fine-grained visual knowledge to Vision-Language Models
  3. 2026Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuning
  4. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models
  5. 2026Remember-R1: Mitigating Long-Context Visual Forgetting through Reinforcement Learning
  6. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Models
  7. 2024Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation
  8. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data