mlsanityData to 7 Oct 2026

ChatRex: Taming Multimodal LLM for Joint Perception and Understanding

arXiv 2411.18363 ↗First posted 27 Nov 2024cs.CVcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Perception and understanding are two pillars of computer vision. While multimodal large language models (MLLM) have demonstrated remarkable visual understanding capabilities, they arguably lack accurate perception abilities, e.g. the stage-of-the-art model Qwen2-VL only achieves a 43.9 recall rate on the COCO dataset, limiting many tasks requiring the combination of perception and understanding. In this work, we aim to bridge this perception gap from both model designing and data development perspectives. We first introduce ChatRex, an MLLM with a decoupled perception design. Instead of having the LLM directly predict box coordinates, we feed the output boxes from a universal proposal network into the LLM, allowing it to output the corresponding box indices to represent its detection results, turning the regression task into a retrieval-based task that LLM handles more proficiently. From the data perspective, we build a fully automated data engine and construct the Rexverse-2M dataset which possesses multiple granularities to support the joint training of perception and understanding. After a three-stage training approach, ChatRex demonstrates strong perception and understanding performance, and the combination of these two capabilities also unlocks many attractive applications, demonstrating their complementary roles in MLLM. Code is available at https://github.com/IDEA-Research/ChatRex.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 94 of its 108 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

94 of 108 references.

Closest ancestors · 30

Same direction, newest first

  1. 2024Pixtral 12B9 Oct
  2. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning30 Sep
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024NVLM: Open Frontier-Class Multimodal LLMs17 Sep
  5. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  6. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models16 Aug
  7. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  8. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  9. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning25 Jun
  10. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  11. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks12 Jun
  12. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
18 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  3. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  5. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models5 Mar
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  7. 2023Osprey: Pixel Understanding with Visual Instruction Tuning15 Dec
  8. 2023VILA: On Pre-training for Visual Language Models12 Dec
  9. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  10. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  11. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V17 Oct
  12. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  13. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  14. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  15. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  16. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  17. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  18. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2023Segment Anythingcited by 8,158
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  13. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  14. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  15. 2021Grounded Language-Image Pre-trainingcited by 1,147
  16. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  17. 2016Modeling Context in Referring Expressionscited by 1,054
  18. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  19. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  20. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  21. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  22. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  23. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  24. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  25. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  26. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  27. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  28. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  29. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  30. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  31. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  32. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  33. 2018CrowdHuman: A Benchmark for Detecting Human in a Crowdcited by 283
  34. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  35. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  36. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  37. 2023PACO: Parts and Attributes of Common Objectscited by 146
  38. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 11.

30 papers.

Sort:

By area:

  1. 2025Detect Anything via Next Point Predictioncited by 66
  2. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  3. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
  4. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  5. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 37
  6. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 23
  7. 2025Referring to Any Personcited by 20
  8. 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
  9. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMscited by 12
  10. 2025WeDetect: Fast Open-Vocabulary Object Detection as Retrievalcited by 11
  11. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  12. 2025EagleVision: Object-level Attribute Multimodal LLM for Remote Sensingcited by 8
18 more
  1. 2025MedSG-Bench: A Benchmark for Medical Image Sequences Groundingcited by 8
  2. 2025VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Predictioncited by 6
  3. 2026Vision as Unified Multimodal Generationcited by 5
  4. 2025Grounding Everything in Tokens for Multimodal Large Language Modelscited by 2
  5. 2025CQ-DINO: Mitigating Gradient Dilution via Category Queries for Vast Vocabulary Object Detectioncited by 1
  6. 2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
  7. 2025CoT4Det: A Chain-of-Thought Framework for Perception-Oriented Vision-Language Taskscited by 1
  8. 2025ZERO: Industry-ready Vision Foundation Model with Multi-modal Promptscited by 0
  9. 2025RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehensioncited by 0
  10. 2026Perception-Aware Multimodal Spatial Reasoning from Monocular Imagescited by 0
  11. 2026Robust Grounding with MLLMs Against Occlusion and Small Objects via Language-Guided Semantic Cuescited by 0
  12. 2026SceneParser: Hierarchical Scene Parsing for Visual Semantics Understandingcited by 0
  13. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  14. 2026HKVLM: Faithful Query--Region Binding for Frozen-Detector Visual Groundingcited by 0
  15. 2026Foundation-Assisted Active Learning for Object Detection Annotationcited by 0
  16. 2026Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Groundingcited by 0
  17. 2026Puffin-World: Scaling a Unified Multimodal Model with Native 3D World Statescited by 0
  18. 2026From Coordinates to Candidate Regions: Temporal Change Localization via Region Selection in Remote Sensing Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
  2. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models
  3. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
  4. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception
  5. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation
  6. 2025Advancing Visual Large Language Model for Multi-granular Versatile Perception
  7. 2026Seeing Right but Saying Wrong: Inter- and Intra-Layer Refinement in MLLMs without Training
  8. 2024MR-MLLM: Mutual Reinforcement of Multimodal Comprehension and Vision Perception