mlsanityData to 14 Oct 2026

Towards Understanding How Knowledge Evolves in Large Vision-Language Models

arXiv 2504.02862 ↗First posted 31 Mar 2025cs.CV cs.LGcited by 10

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large Vision-Language Models (LVLMs) are gradually becoming the foundation for many artificial intelligence applications. However, understanding their internal working mechanisms has continued to puzzle researchers, which in turn limits the further enhancement of their capabilities. In this paper, we seek to investigate how multimodal knowledge evolves and eventually induces natural languages in LVLMs. We design a series of novel strategies for analyzing internal knowledge within LVLMs, and delve into the evolution of multimodal knowledge from three levels, including single token probabilities, token probability distributions, and feature encodings. In this process, we identify two key nodes in knowledge evolution: the critical layers and the mutation layers, dividing the evolution process into three stages: rapid evolution, stabilization, and mutation. Our research is the first to reveal the trajectory of knowledge evolution in LVLMs, providing a fresh perspective for understanding their underlying mechanisms. Our codes are available at https://github.com/XIAO4579/Vlm-interpretability.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 53 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

47 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
34 more
  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  2. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  4. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  6. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  7. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  9. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  10. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  11. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  13. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  14. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  15. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  16. 2022Confident Adaptive Language Modelingcited by 310
  17. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  18. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  19. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  20. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  21. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  22. 2019Depth-Adaptive Transformercited by 236
  23. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  24. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  25. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  26. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  27. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  28. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  29. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 115
  30. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  31. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  32. 2021FrameExit: Conditional Early Exiting for Efficient Video Recognitioncited by 59
  33. 2017Early Stopping without a Validation Setcited by 39
  34. 2024Libra: Building Decoupled Vision System on Large Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 35
  2. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 13
  3. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Modelscited by 7
  4. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  5. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 3
  6. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  7. 2026Diagnosing Visual Ignorance in Vision-Language Modelscited by 2
  8. 2026Hallucination-aware intermediate representation edit in large vision-language modelscited by 0
  9. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0
  10. 2026Neuromorphic Object Detection: An In-Depth Study and Future Directionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SE-VLN: A Self-Evolving Vision-Language Navigation Framework Based on Multimodal Large Language Models
  2. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
  3. 2026Innovator-VL: A Multimodal Large Language Model for Scientific Discovery
  4. 2025When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations
  5. 2026RISE: Reliable Improvement in Self-Evolving Vision-Language Models
  6. 2024EvoWiki: Evaluating LLMs on Evolving Knowledge
  7. 2024Bias Amplification in Language Model Evolution: An Iterated Learning Perspective
  8. 2026RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World