# ParGo: Bridging Vision-Language with Partial and Global Views

arXiv: https://arxiv.org/abs/2408.12928 · HTML: https://mlsanity.com/paper/2408.12928/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [Honeybee: Locality-enhanced Projector for Multimodal LLM](https://mlsanity.com/paper/2312.06742/) (2023)
- [BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions](https://mlsanity.com/paper/2308.09936/) (2023)
- [The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World](https://mlsanity.com/paper/2308.01907/) (2023)
- [CapsFusion: Rethinking Image-Text Data at Scale](https://mlsanity.com/paper/2310.20550/) (2023)
- [The All-Seeing Project V2: Towards General Relation Comprehension of the Open World](https://mlsanity.com/paper/2402.19474/) (2024)

## Built on by (most cited first, 30 of 40)

- [LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token](https://mlsanity.com/paper/2501.03895/) (2025)
- [Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidance](https://mlsanity.com/paper/2412.12974/) (2024)
- [ViSpeak: Visual Instruction Feedback in Streaming Videos](https://mlsanity.com/paper/2503.12769/) (2025)
- [ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations](https://mlsanity.com/paper/2501.14607/) (2025)
- [WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?](https://mlsanity.com/paper/2505.11015/) (2025)
- [Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network](https://mlsanity.com/paper/2412.15678/) (2024)
- [LangBridge: Interpreting Image as a Combination of Language Embeddings](https://mlsanity.com/paper/2503.19404/) (2025)
- [Connector-S: A Survey of Connectors in Multi-modal Large Language Models](https://mlsanity.com/paper/2502.11453/) (2025)
- [Panoptic Captioning: An Equivalence Bridge for Image and Text](https://mlsanity.com/paper/2505.16334/) (2025)
- [A Comprehensive Review of AI Agents: Transforming Possibilities in Technology and Beyond](https://mlsanity.com/paper/2508.11957/) (2025)
- [EvoPrune: Early-Stage Visual Token Pruning for Efficient MLLMs](https://mlsanity.com/paper/2603.03681/) (2026)
- [Task-Oriented 6-DoF Grasp Pose Detection in Clutters](https://mlsanity.com/paper/2502.16976/) (2025)
- [ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models](https://mlsanity.com/paper/2509.24837/) (2025)
- [TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generation](https://mlsanity.com/paper/2503.17669/) (2025)
- [Hear the Scene: Audio-Enhanced Text Spotting](https://mlsanity.com/paper/2412.19504/) (2024)
- [First-place Solution for Streetscape Shop Sign Recognition Competition](https://mlsanity.com/paper/2501.02811/) (2025)
- [Cross-Modal Synergies: Unveiling the Potential of Motion-Aware Fusion Networks in Handling Dynamic and Static ReID Scenarios](https://mlsanity.com/paper/2502.00665/) (2025)
- [OMR-Diffusion:Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Intent Understanding](https://mlsanity.com/paper/2503.17660/) (2025)
- [LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Data](https://mlsanity.com/paper/2601.09118/) (2026)
- [Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Method](https://mlsanity.com/paper/2601.09238/) (2026)
- [MA-LipNet: Multi-Dimensional Attention Networks for Robust Lipreading](https://mlsanity.com/paper/2601.20881/) (2026)
- [The Diffusion Duet: Harmonizing Dual Channels with Wavelet Suppression for Image Separation](https://mlsanity.com/paper/2602.13361/) (2026)
- [Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignment](https://mlsanity.com/paper/2602.15903/) (2026)
- [Efficient Image Super-Resolution with Multi-Scale Spatial Adaptive Attention Networks](https://mlsanity.com/paper/2603.00118/) (2026)
- [DQE-CIR: Distinctive Query Embeddings through Learnable Attribute Weights and Target Relative Negative Sampling in Composed Image Retrieval](https://mlsanity.com/paper/2603.04037/) (2026)
- [DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding](https://mlsanity.com/paper/2603.11380/) (2026)
- [A Deformable Attention-Based Detection Transformer with Cross-Scale Feature Fusion for Industrial Coil Spring Inspection](https://mlsanity.com/paper/2603.13429/) (2026)
- [Transmittance-Guided Structure-Texture Decomposition for Nighttime Image Dehazing](https://mlsanity.com/paper/2603.29507/) (2026)
- [Adaptive Slicing-Assisted Hyper Inference for Enhanced Small Object Detection in High-Resolution Imagery](https://mlsanity.com/paper/2604.19233/) (2026)
- [Lightweight Real-Time Rendering Parameter Optimization via XGBoost-Driven Lookup Tables](https://mlsanity.com/paper/2604.25178/) (2026)

## Abstract (arXiv)

This work presents ParGo, a novel Partial-Global projector designed to connect the vision and language modalities for Multimodal Large Language Models (MLLMs). Unlike previous works that rely on global attention-based projectors, our ParGo bridges the representation gap between the separately pre-trained vision encoders and the LLMs by integrating global and partial views, which alleviates the overemphasis on prominent regions. To facilitate the effective training of ParGo, we collect a large-scale detail-captioned image-text dataset named ParGoCap-1M-PT, consisting of 1 million images paired with high-quality captions. Extensive experiments on several MLLM benchmarks demonstrate the effectiveness of our ParGo, highlighting its superiority in aligning vision and language modalities. Compared to conventional Q-Former projector, our ParGo achieves an improvement of 259.96 in MME benchmark. Furthermore, our experiments reveal that ParGo significantly outperforms other projectors, particularly in tasks that emphasize detail perception ability.