mlsanityData to 14 Oct 2026

PerceptionGPT: Effectively Fusing Visual Perception into LLM

arXiv 2311.06612 ↗First posted 11 Nov 2023cs.CV cs.CLcited by 59

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The integration of visual inputs with large language models (LLMs) has led to remarkable advancements in multi-modal capabilities, giving rise to visual large language models (VLLMs). However, effectively harnessing VLLMs for intricate visual perception tasks remains a challenge. In this paper, we present a novel end-to-end framework named PerceptionGPT, which efficiently and effectively equips the VLLMs with visual perception abilities by leveraging the representation power of LLMs' token embedding. Our proposed method treats the token embedding of the LLM as the carrier of spatial information, then leverage lightweight visual task encoders and decoders to perform visual perception tasks (e.g., detection, segmentation). Our approach significantly alleviates the training difficulty suffered by previous approaches that formulate the visual outputs as discrete tokens, and enables achieving superior performance with fewer trainable parameters, less training data and shorted training time. Moreover, as only one token embedding is required to decode the visual outputs, the resulting sequence length during inference is significantly reduced. Consequently, our approach enables accurate and flexible representations, seamless integration of visual perception tasks, and efficient handling of a multiple of visual outputs. We validate the effectiveness and efficiency of our approach through extensive experiments. The results demonstrate significant improvements over previous methods with much fewer trainable parameters and GPU hours, which facilitates future research in enabling LLMs with visual perception abilities.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 11 Nov 2023.

Builds on

46 of 55 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
34 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  7. 2021Grounded Language-Image Pre-trainingcited by 1,156
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  13. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  14. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  15. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 620
  16. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  17. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  18. 2023Segment Everything Everywhere All at Oncecited by 548
  19. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  20. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  22. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  23. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
  24. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  25. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  26. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  27. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  28. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  29. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 139
  30. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 50
  31. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  32. 2021G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitationcited by 28
  33. 2022Artificial Intelligence for the Metaverse: A Surveycited by 25
  34. 2021Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillationcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 17; 2025: 24; 2026: 16.

59 papers.

Sort:

By area:

  1. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
  2. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  3. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 76
  4. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 71
  5. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 69
  6. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 65
  7. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  8. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 53
  9. 2024Graphic Design with Large Multimodal Modelcited by 40
  10. 2025DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Responsecited by 36
  11. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  12. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
38 more
  1. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 34
  2. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  3. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  4. 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learningcited by 31
  5. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  6. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 30
  7. 2023Lenna: Language Enhanced Reasoning Detection Assistantcited by 29
  8. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  9. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  10. 2024SegLLM: Multi-round Reasoning Segmentationcited by 25
  11. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  12. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  13. 2025Multimodal Referring Segmentation: A Surveycited by 22
  14. 2025Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Datacited by 8
  15. 2024LLMI3D: MLLM-based 3D Perception from a Single 2D Imagecited by 7
  16. 2025UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluationcited by 7
  17. 2025MR. Judge: Multimodal Reasoner as a Judgecited by 7
  18. 2025DisTime: Distribution-based Time Representation for Video Large Language Modelscited by 7
  19. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  20. 2024KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehensioncited by 5
  21. 2025CoT Referring: Improving Referring Expression Tasks with Grounded Reasoningcited by 5
  22. 2026Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Groundingcited by 5
  23. 2025Ground-V: Teaching VLMs to Ground Complex Instructions in Pixelscited by 4
  24. 2025AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agentscited by 4
  25. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustnesscited by 3
  26. 2025MINGLE: VLMs for Semantically Complex Region Detection in Urban Scenescited by 3
  27. 2026DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Modelscited by 3
  28. 2026Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generationcited by 3
  29. 2025Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Drivingcited by 2
  30. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Modelscited by 2
  31. 2026Multi-scale hypergraph meets LLMs: Aligning large language models for time series analysiscited by 2
  32. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  33. 2026From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Groundingcited by 2
  34. 2025EVTP-IVS: Effective Visual Token Pruning For Unifying Instruction Visual Segmentation In Multi-Modal Large Language Modelscited by 1
  35. 2025Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentationcited by 1
  36. 2025MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Imagescited by 1
  37. 2026A Technical Taxonomy of LLM Agent Communication Protocolscited by 1
  38. 2025KptLLM++: Towards Generic Keypoint Comprehension with Large Language Modelcited by 0

50 most cited of 59 All 59 (JSON)