mlsanityData to 7 Oct 2026

NExT-Chat: An LMM for Chat, Detection and Segmentation

arXiv 2311.04498 ↗First posted 8 Nov 2023cs.CV cs.AI cs.CLcited by 79

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to enhance the level of visual comprehension, recent studies have equipped LMMs with region-level understanding capabilities by representing object bounding box coordinates as a series of text sequences (pix2seq). In this paper, we introduce a novel paradigm for object location modeling called pix2emb method, where we ask the LMM to output the location embeddings and then decode them with different decoders. This paradigm allows us to use different location formats (such as bounding boxes and masks) in multimodal conversations. Leveraging the proposed pix2emb method, we train an LMM named NExT-Chat and demonstrate its capability of handling multiple tasks like visual grounding, region captioning, and grounded reasoning. Comprehensive experiments show the effectiveness of our NExT-Chat on various tasks, e.g., NExT-Chat (87.7) vs. Shikra (86.9) on POPE-Random, NExT-Chat (68.9) vs. LISA (67.9) on referring expression segmentation task, and NExT-Chat (79.6) vs. Kosmos-2 (62.3) on region caption task. The code and model are released at https://github.com/NExT-ChatV/NExT-Chat.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 49 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

42 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  3. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models23 May
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  5. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
25 more
  1. 2016Modeling Context in Referring Expressionscited by 1,054
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  6. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  10. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  11. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  12. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  13. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  14. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  15. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  16. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  17. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  18. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  19. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  20. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  21. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  22. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  23. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  24. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 79
  25. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 35; 2025: 26; 2026: 13.

79 papers.

Sort:

By area:

  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  2. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  3. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  4. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  5. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  6. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  7. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  8. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  9. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  10. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  11. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  12. 2024Toward Visual Grounding: A Surveycited by 52
38 more
  1. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  2. 2024GeoGround: A Unified Large Vision-Language Model for Remote Sensing Visual Groundingcited by 46
  3. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  4. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  5. 2024On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detectioncited by 40
  6. 2025SegEarth-R1: Geospatial Pixel Reasoning via Large Language Modelcited by 40
  7. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  8. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  9. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  10. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  11. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perceptioncited by 32
  12. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Expertscited by 29
  13. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  14. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  15. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoningcited by 27
  16. 2024SegLLM: Multi-round Reasoning Segmentationcited by 25
  17. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Modelcited by 24
  18. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  19. 2025Multimodal Referring Segmentation: A Surveycited by 22
  20. 2024Imp: Highly Capable Large Multimodal Models for Mobile Devicescited by 19
  21. 2024From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Informationcited by 16
  22. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 14
  23. 2023VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generationcited by 12
  24. 2024AesopAgent: Agent-driven Evolutionary System on Story-to-Video Productioncited by 12
  25. 2025Visual Intention Grounding for Egocentric Assistantscited by 11
  26. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Modelscited by 10
  27. 2025UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenescited by 10
  28. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  29. 2023Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Modelscited by 8
  30. 2025MedSG-Bench: A Benchmark for Medical Image Sequences Groundingcited by 8
  31. 2025CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videoscited by 8
  32. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 8
  33. 2024LLMI3D: MLLM-based 3D Perception from a Single 2D Imagecited by 7
  34. 2024HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understandingcited by 7
  35. 2025Think Before You Segment: High-Quality Reasoning Segmentation with GPT Chain of Thoughtscited by 7
  36. 2026RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMscited by 7
  37. 2024Localizing Events in Videos with Multimodal Queriescited by 6
  38. 2024Spider: Any-to-Many Multimodal LLMcited by 6

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024F-LMM: Grounding Frozen Large Multimodal Models
  2. 2024ChatterBox: Multi-round Multimodal Referring and Grounding
  3. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervision
  4. 2023GLaMM: Pixel Grounding Large Multimodal Model
  5. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
  6. 2025Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding
  7. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  8. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representations