mlsanityData to 7 Oct 2026

Pix2seq: A Language Modeling Framework for Object Detection

arXiv 2109.10852 ↗First posted 22 Sep 2021cs.CV cs.AI cs.CLcited by 345

Not on the map. Why?

Cited by 83 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection as a language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed as sequences of discrete tokens, and we train a neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations, our approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms.

Key facts. Cited by 345 papers in the map (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

43 of 61 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2017Mask R-CNNcited by 7,259
31 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2015Fast R-CNNcited by 2,930
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  11. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  13. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  14. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  15. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  16. 2019Objects as Pointscited by 1,172
  17. 2016Deep Networks with Stochastic Depthcited by 973
  18. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  19. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  20. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  21. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  22. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380
  23. 2015End-to-end people detection in crowded scenescited by 140
  24. 2016Fully Convolutional Instance-aware Semantic Segmentationcited by 122
  25. 2014Grammar as a Foreign Languagecited by 70
  26. 2018Retina U-Net: Embarrassingly Simple Exploitation of Segmentation Supervision for Medical Object Detectioncited by 46
  27. 2013Some Improvements on Deep Convolutional Neural Network Based Image Classificationcited by 34
  28. 2013Scalable Object Detection using Deep Neural Networkscited by 30
  29. 2015Recurrent Instance Segmentationcited by 21
  30. 2017Recurrent Neural Networks for Semantic Instance Segmentationcited by 11
  31. 2015Learning to decompose for object detection and instance segmentationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 52; 2023: 100; 2024: 88; 2025: 44; 2026: 53.

345 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  5. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2022A Generalist Agentcited by 802
  8. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  11. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
38 more
  1. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  2. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  3. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  4. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  5. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  6. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  7. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  8. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  9. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  10. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  11. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  12. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  13. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  14. 2023Towards Generalist Biomedical AIcited by 286
  15. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  16. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  17. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  18. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  19. 2023SegGPT: Segmenting Everything In Contextcited by 193
  20. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  21. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  22. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  23. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  24. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  25. 2022A Unified Sequence Interface for Vision Taskscited by 152
  26. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  27. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  28. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  29. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  30. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  31. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  32. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  33. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  34. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  35. 2022Backbone is All Your Need: A Simplified Architecture for Visual Object Trackingcited by 119
  36. 20234M: Massively Multimodal Masked Modelingcited by 118
  37. 2021A Survey of Visual Transformerscited by 117
  38. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117

50 most cited of 345 All 345 (JSON)