mlsanityData to 14 Oct 2026

VisionPangu: A Compact and Fine-Grained Multimodal Assistant with 1.7B Parameters

arXiv 2603.04957 ↗First posted 5 Mar 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multimodal Models (LMMs) have achieved strong performance in vision-language understanding, yet many existing approaches rely on large-scale architectures and coarse supervision, which limits their ability to generate detailed image captions. In this work, we present VisionPangu, a compact 1.7B-parameter multimodal model designed to improve detailed image captioning through efficient multimodal alignment and high-quality supervision. Our model combines an InternVL-derived vision encoder with the OpenPangu-Embedded language backbone via a lightweight MLP projector and adopts an instruction-tuning pipeline inspired by LLaVA. By incorporating dense human-authored descriptions from the DOCCI dataset, VisionPangu improves semantic coherence and descriptive richness without relying on aggressive model scaling. Experimental results demonstrate that compact multimodal models can achieve competitive performance while producing more structured and detailed captions. The code and model weights will be publicly available at https://www.modelscope.cn/models/asdfgh007/visionpangu.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 9 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

8 of 9 references.

References · 8

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  8. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 62

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
  2. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  3. 2024Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  6. 2025Visual Representation Alignment for Multimodal Large Language Models
  7. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
  8. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding