mlsanityData to 7 Oct 2026

ParGo: Bridging Vision-Language with Partial and Global Views

arXiv 2408.12928 ↗First posted 23 Aug 2024cs.CVcited by 40

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This work presents ParGo, a novel Partial-Global projector designed to connect the vision and language modalities for Multimodal Large Language Models (MLLMs). Unlike previous works that rely on global attention-based projectors, our ParGo bridges the representation gap between the separately pre-trained vision encoders and the LLMs by integrating global and partial views, which alleviates the overemphasis on prominent regions. To facilitate the effective training of ParGo, we collect a large-scale detail-captioned image-text dataset named ParGoCap-1M-PT, consisting of 1 million images paired with high-quality captions. Extensive experiments on several MLLM benchmarks demonstrate the effectiveness of our ParGo, highlighting its superiority in aligning vision and language modalities. Compared to conventional Q-Former projector, our ParGo achieves an improvement of 259.96 in MME benchmark. Furthermore, our experiments reveal that ParGo significantly outperforms other projectors, particularly in tasks that emphasize detail perception ability.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 53 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

36 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023CapsFusion: Rethinking Image-Text Data at Scale31 Oct
  2. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions19 Aug
  3. 2022ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation9 Nov
  4. 2022ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training30 Sep
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
19 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  9. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  12. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  13. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  14. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  15. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  16. 2023SPTS v2: Single-Point Scene Text Spottingcited by 66
  17. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  18. 2023Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizercited by 53
  19. 2024XOV-Action: Towards Generalizable Open-Vocabulary Action Recognitioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 23.

40 papers.

Sort:

By area:

  1. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  2. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  3. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  4. 2025ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundationscited by 30
  5. 2025WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?cited by 26
  6. 2024Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Networkcited by 15
  7. 2025LangBridge: Interpreting Image as a Combination of Language Embeddingscited by 9
  8. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  9. 2025Panoptic Captioning: An Equivalence Bridge for Image and Textcited by 6
  10. 2025A Comprehensive Review of AI Agents: Transforming Possibilities in Technology and Beyondcited by 6
  11. 2026EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMscited by 5
  12. 2025Task-Oriented 6-DoF Grasp Pose Detection in Clutterscited by 4
28 more
  1. 2025ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Modelscited by 4
  2. 2025TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generationcited by 1
  3. 2024Hear the Scene: Audio-Enhanced Text Spottingcited by 0
  4. 2025First-place Solution for Streetscape Shop Sign Recognition Competitioncited by 0
  5. 2025Cross-Modal Synergies: Unveiling the Potential of Motion-Aware Fusion Networks in Handling Dynamic and Static ReID Scenarioscited by 0
  6. 2025OMR-Diffusion:Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Intent Understandingcited by 0
  7. 2026LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Datacited by 0
  8. 2026Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Methodcited by 0
  9. 2026MA-LipNet: Multi-Dimensional Attention Networks for Robust Lipreadingcited by 0
  10. 2026The Diffusion Duet: Harmonizing Dual Channels with Wavelet Suppression for Image Separationcited by 0
  11. 2026Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignmentcited by 0
  12. 2026Efficient Image Super-Resolution with Multi-Scale Spatial Adaptive Attention Networkscited by 0
  13. 2026DQE-CIR: Distinctive Query Embeddings through Learnable Attribute Weights and Target Relative Negative Sampling in Composed Image Retrievalcited by 0
  14. 2026DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understandingcited by 0
  15. 2026A Deformable Attention-Based Detection Transformer with Cross-Scale Feature Fusion for Industrial Coil Spring Inspectioncited by 0
  16. 2026Transmittance-Guided Structure-Texture Decomposition for Nighttime Image Dehazingcited by 0
  17. 2026Adaptive Slicing-Assisted Hyper Inference for Enhanced Small Object Detection in High-Resolution Imagerycited by 0
  18. 2026Lightweight Real-Time Rendering Parameter Optimization via XGBoost-Driven Lookup Tablescited by 0
  19. 2026Image Classification via Random Dilated Convolution with Multi-Branch Feature Extraction and Context Excitationcited by 0
  20. 2026Gait Recognition via Deep Residual Networks and Multi-Branch Feature Fusioncited by 0
  21. 2026Multi-Branch Non-Homogeneous Image Dehazing via Concentration Partitioning and Image Fusioncited by 0
  22. 2026Local Spatiotemporal Convolutional Network for Robust Gait Recognitioncited by 0
  23. 2026Do You Need Text Rectification? Soft Attention Mask Embedding for Rectification-Free Scene Text Spottingcited by 0
  24. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  25. 2026ProWAFT: A ROMA-LPD Instance for Workload-Aware and Dynamic Fault Tolerance in FPGA-Based CNN Acceleratorscited by 0
  26. 2026Holo-Captioning: Toward the Text Equivalent of 3D Scenescited by 0
  27. 2026Rank-Consistent Set Reasoning for Co-Salient Object Detectioncited by 0
  28. 2026Contour-Guided Spectral Routing for Robust Real-Time Pedestrian Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception
  2. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  3. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  4. 2021FLAVA: A Foundational Language And Vision Alignment Model
  5. 2025Rethinking Visual Information Processing in Multimodal LLMs
  6. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  7. 2025TopoPerception: A Shortcut-Free Evaluation of Global Visual Perception in Large Vision-Language Models
  8. 2024Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models