mlsanityData to 14 Oct 2026

UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling

arXiv 2111.12085 ↗First posted 23 Nov 2021cs.CVcited by 122

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We propose UniTAB that Unifies Text And Box outputs for grounded vision-language (VL) modeling. Grounded VL tasks such as grounded captioning require the model to generate a text description and align predicted words with object regions. To achieve this, models must generate desired text and box outputs together, and meanwhile indicate the alignments between words and boxes. In contrast to existing solutions that use multiple separate modules for different outputs, UniTAB represents both text and box outputs with a shared token sequence, and introduces a special <obj> token to naturally indicate word-box alignments in the sequence. UniTAB thus could provide a more comprehensive and interpretable image description, by freely grounding generated words to object regions. On grounded captioning, UniTAB presents a simpler solution with a single output head, and significantly outperforms state of the art in both grounding and captioning evaluations. On general VL tasks that have different desired output formats (i.e., text, box, or their combination), UniTAB with a single network achieves better or comparable performance than task-specific state of the art. Experiments cover 7 VL benchmarks, including grounded captioning, visual grounding, image captioning, and visual question answering. Furthermore, UniTAB's unified multi-task network and the task-agnostic output sequence design make the model parameter efficient and generalizable to new tasks.

Key facts. Cited by 122 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 71 of its 88 references that are in the map. First posted on arXiv 23 Nov 2021.

Builds on

71 of 88 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  2. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training25 Jun
  3. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning3 Jun
  4. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  5. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  13. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  15. 2016Modeling Context in Referring Expressionscited by 1,063
  16. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  17. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  18. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  19. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  20. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  21. 2018Object Hallucination in Image Captioningcited by 669
  22. 2016Self-critical Sequence Training for Image Captioningcited by 547
  23. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  24. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 472
  25. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  26. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  27. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  28. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  29. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  30. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  31. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  32. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  33. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  34. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  35. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  36. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  37. 2017Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localizationcited by 207
  38. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 13; 2023: 51; 2024: 33; 2025: 16; 2026: 9.

122 papers.

Sort:

By area:

  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  5. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  6. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  7. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  8. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  9. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  10. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 179
  11. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 148
  12. 2024MAIRA-2: Grounded Radiology Report Generationcited by 145
38 more
  1. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  2. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  3. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  4. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  6. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  7. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  8. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  9. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  10. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
  11. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  12. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 80
  13. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  14. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  15. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  16. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  17. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  18. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  19. 2023Towards Unified Token Learning for Vision-Language Trackingcited by 62
  20. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
  21. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 54
  22. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 53
  23. 2024Toward Visual Grounding: A Surveycited by 52
  24. 2023VIP5: Towards Multimodal Foundation Models for Recommendationcited by 46
  25. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  26. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  27. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  28. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 38
  29. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  30. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  31. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Modelscited by 31
  32. 2024HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Groundingcited by 31
  33. 2024Language-Image Models with 3D Understandingcited by 31
  34. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Modelscited by 30
  35. 2022Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Groundingcited by 29
  36. 2023Lenna: Language Enhanced Reasoning Detection Assistantcited by 29
  37. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  38. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022UNIMO-2: End-to-End Unified Vision-Language Grounded Learning
  2. 2024The ART of Composition: Attention-Regularized Training for Compositional Visual Grounding
  3. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  4. 2023GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regions
  5. 2026GAVEL: Grounded Caption Error Verification and Localization
  6. 2024Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM
  7. 2025Discovering Meaningful Units with Visually Grounded Semantics from Image Captions
  8. 2026Vision-Language Grounding as Bidirectional Concept Correspondence