mlsanityData to 7 Oct 2026

GRiT: A Generative Region-to-text Transformer for Object Understanding

arXiv 2212.00280 ↗First posted 1 Dec 2022cs.CVcited by 119

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a Generative RegIon-to-Text transformer, GRiT, for object understanding. The spirit of GRiT is to formulate object understanding as <region, text> pairs, where region locates objects and text describes objects. For example, the text in object detection denotes class names while that in dense captioning refers to descriptive sentences. Specifically, GRiT consists of a visual encoder to extract image features, a foreground object extractor to localize objects, and a text decoder to generate open-set object descriptions. With the same model architecture, GRiT can understand objects via not only simple nouns, but also rich descriptive sentences including object attributes or actions. Experimentally, we apply GRiT to object detection and dense captioning tasks. GRiT achieves 60.4 AP on COCO 2017 test-dev for object detection and 15.5 mAP on Visual Genome for dense captioning. Code is available at https://github.com/JialianW/GRiT

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

38 of 51 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2017Deformable Convolutional Networkscited by 1,481
26 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  3. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2019Objects as Pointscited by 1,172
  6. 2021Grounded Language-Image Pre-trainingcited by 1,147
  7. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  10. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  11. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  12. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  13. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  14. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  15. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  16. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  17. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  18. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 315
  19. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  20. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  21. 2021Dynamic Head: Unifying Object Detection Heads with Attentionscited by 185
  22. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  23. 2021Probabilistic two-stage detectioncited by 130
  24. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labelscited by 71
  25. 2016Dense Captioning with Joint Inference and Visual Contextcited by 40
  26. 2019Context and Attribute Grounded Dense Captioningcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 43; 2024: 40; 2025: 29; 2026: 7.

119 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  6. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  7. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  9. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  10. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  11. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  12. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
38 more
  1. 2023Video Understanding with Large Language Models: A Surveycited by 203
  2. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  3. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  4. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  5. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  6. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  7. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  8. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  9. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  10. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  11. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  12. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  13. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  14. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 93
  15. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  16. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  17. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
  18. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 85
  19. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 84
  20. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  21. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  22. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79
  23. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  24. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  25. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  26. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  27. 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 53
  28. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53
  29. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  30. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  31. 2024DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detectioncited by 48
  32. 2024Hawk: Learning to Understand Open-World Video Anomaliescited by 47
  33. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 47
  34. 2026Ultralytics YOLO26: Unified Real-Time End-to-End Vision Modelscited by 47
  35. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  36. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 42
  37. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  38. 2023Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generationcited by 38

50 most cited of 119 All 119 (JSON)