mlsanityData to 7 Oct 2026

Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

arXiv 2306.15195 ↗First posted 27 Jun 2023cs.CVcited by 891

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In human conversations, individuals can indicate relevant regions within a scene while addressing others. In turn, the other person can then respond by referring to specific regions if necessary. This natural referential ability in dialogue remains absent in current Multimodal Large Language Models (MLLMs). To fill this gap, this paper proposes an MLLM called Shikra, which can handle spatial coordinate inputs and outputs in natural language. Its architecture consists of a vision encoder, an alignment layer, and a LLM. It is designed to be straightforward and simple, without the need for extra vocabularies, position encoder, pre-/post-detection modules, or external plug-in models. All inputs and outputs are in natural language form. Referential dialogue is a superset of various vision-language (VL) tasks. Shikra can naturally handle location-related tasks like REC and PointQA, as well as conventional VL tasks such as Image Captioning and VQA. Experimental results showcase Shikra's promising performance. Furthermore, it enables numerous exciting applications, like providing mentioned objects' coordinates in chains of thoughts and comparing user-pointed regions similarities. Our code, model and dataset are accessed at https://github.com/shikras/shikra.

Key facts. Cited by 891 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 57 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

47 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  4. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 43

  1. 2023Segment Anythingcited by 8,158
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2015Fast R-CNNcited by 2,930
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
31 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  11. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  12. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  13. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  14. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  15. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  17. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  18. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  19. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  20. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  21. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  22. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  23. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  24. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  25. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  26. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  27. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  28. 2020End-to-End Object Detection with Fully Convolutional Networkcited by 70
  29. 2020More Grounded Image Captioning by Distilling Image-Text Matching Modelcited by 43
  30. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39
  31. 2023DisCLIP: Open-Vocabulary Referring Expression Generationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 109; 2024: 379; 2025: 243; 2026: 160.

891 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
38 more
  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  4. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  7. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  8. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  12. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  13. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  14. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  15. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  16. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  17. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  18. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  19. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  20. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  21. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  22. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  23. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  24. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  25. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  26. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  27. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  28. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  29. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  30. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  31. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  32. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  33. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  34. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  35. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  36. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  37. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  38. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247

50 most cited of 891 All 891 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding
  2. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  3. 2024Personalized Large Vision-Language Models
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  6. 2025Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning
  7. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  8. 2023Enhancing the Spatial Awareness Capability of Multi-Modal Large Language Model