mlsanityData to 7 Oct 2026

InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

arXiv 2401.16420 ↗First posted 29 Jan 2024cs.CV cs.CLcited by 326

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension. This model goes beyond conventional vision-language understanding, adeptly crafting interleaved text-image content from diverse inputs like outlines, detailed textual specifications, and reference images, enabling highly customizable content creation. InternLM-XComposer2 proposes a Partial LoRA (PLoRA) approach that applies additional LoRA parameters exclusively to image tokens to preserve the integrity of pre-trained language knowledge, striking a balance between precise vision understanding and text composition with literary talent. Experimental results demonstrate the superiority of InternLM-XComposer2 based on InternLM2-7B in producing high-quality long-text multi-modal content and its exceptional vision-language understanding performance across various benchmarks, where it not only significantly outperforms existing multimodal models but also matches or even surpasses GPT-4V and Gemini Pro in certain assessments. This highlights its remarkable proficiency in the realm of multimodal understanding. The InternLM-XComposer2 model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer.

Key facts. Cited by 326 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 76 of its 92 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

76 of 92 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  2. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  5. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  6. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  7. 2023MLLM-DataEngine: An Iterative Refinement Approach for MLLM25 Aug
  8. 2023VIGC: Visual Instruction Generation and Correction24 Aug
  9. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models21 Aug
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
4 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2024Mixtral of Expertscited by 1,785
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2021Grounded Language-Image Pre-trainingcited by 1,147
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  14. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  16. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  17. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  18. 2016A Diagram Is Worth A Dozen Imagescited by 793
  19. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  20. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  21. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  22. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  23. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  24. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  25. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  26. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  27. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  28. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  29. 2018nocaps: novel object captioning at scalecited by 444
  30. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  31. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  32. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  33. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  34. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  35. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  36. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  37. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  38. 2023Emu: Generative Pretraining in Multimodalitycited by 156

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 176; 2025: 118; 2026: 32.

326 papers.

Sort:

By area:

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  5. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  6. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  7. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  8. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  9. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  10. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  11. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  12. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
38 more
  1. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  2. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  3. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  4. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  5. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  6. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  7. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  8. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  9. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  10. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  11. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  12. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  13. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  14. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  15. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  16. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  17. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  18. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  19. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  20. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  21. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  22. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  23. 2024IMAGDressing-v1: Customizable Virtual Dressingcited by 100
  24. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  25. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  26. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  27. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  28. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  29. 2024Towards Open-ended Visual Quality Comparisoncited by 85
  30. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  31. 2024Benchmarking and Improving Detail Image Captioncited by 81
  32. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  33. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  34. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  35. 2024A Survey on LoRA of Large Language Modelscited by 67
  36. 2024Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Comparecited by 66
  37. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  38. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 63

50 most cited of 326 All 326 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  2. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  3. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training
  4. 2025ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
  7. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  8. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks