mlsanityData to 7 Oct 2026

X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks

arXiv 2204.05626 ↗First posted 12 Apr 2022cs.CV cs.CL cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the challenging instance-wise vision-language tasks, where the free-form language is required to align with the objects instead of the whole image. To address these tasks, we propose X-DETR, whose architecture has three major components: an object detector, a language encoder, and vision-language alignment. The vision and language streams are independent until the end and they are aligned using an efficient dot-product operation. The whole network is trained end-to-end, such that the detector is optimized for the vision-language tasks instead of an off-the-shelf component. To overcome the limited size of paired object-language annotations, we leverage other weak types of supervision to expand the knowledge coverage. This simple yet effective architecture of X-DETR shows good accuracy and fast speeds for multiple instance-wise vision-language tasks, e.g., 16.4 AP on LVIS detection of 1.2K categories at ~20 frames per second without using any LVIS annotation during training.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 52 of its 65 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

52 of 65 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2017Mask R-CNNcited by 7,259
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  2. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  11. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  13. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  14. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  15. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  16. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  18. 2021Grounded Language-Image Pre-trainingcited by 1,147
  19. 2016Modeling Context in Referring Expressionscited by 1,054
  20. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  21. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  22. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  23. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  24. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  25. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  26. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  27. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  28. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  29. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  30. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  31. 2019Connecting Vision and Language with Localized Narrativescited by 225
  32. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  33. 2018Zero-Shot Object Detectioncited by 189
  34. 2015Natural Language Object Retrievalcited by 124
  35. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  36. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  37. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  38. 2018Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Conceptscited by 48

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 20; 2024: 10; 2025: 1; 2026: 1.

41 papers.

Sort:

By area:

  1. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  2. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  3. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  4. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  5. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  6. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  7. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  8. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 95
  9. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  10. 2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
  11. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 66
  12. 2023Object-Aware Distillation Pyramid for Open-Vocabulary Object Detectioncited by 63
29 more
  1. 2024DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detectioncited by 48
  2. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
  3. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptorscited by 39
  4. 2024Generative Region-Language Pretraining for Open-Ended Object Detectioncited by 29
  5. 2024AIDE: An Automatic Data Engine for Object Detection in Autonomous Drivingcited by 26
  6. 2023DesCo: Learning Object Recognition with Rich Language Descriptionscited by 25
  7. 2023Taming Self-Training for Open-Vocabulary Object Detectioncited by 25
  8. 2024Textual Query-Driven Mask Transformer for Domain Generalized Segmentationcited by 20
  9. 2023OmniLabel: A Challenging Benchmark for Language-Based Object Detectioncited by 16
  10. 2023ScaleDet: A Scalable Multi-Dataset Object Detectorcited by 16
  11. 2023How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detectioncited by 15
  12. 2022Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Taskscited by 12
  13. 2023Learning to Name Classes for Vision and Language Modelscited by 12
  14. 2023ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detectioncited by 12
  15. 2024Unifying Global and Local Scene Entities Modelling for Precise Action Spottingcited by 8
  16. 2023Z-GMOT: Zero-shot Generic Multiple Object Trackingcited by 7
  17. 2023Open-Vocabulary Object Detection via Scene Graph Discoverycited by 7
  18. 2023Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detectioncited by 7
  19. 2023Object-Centric Open-Vocabulary Image-Retrieval with Aggregated Featurescited by 5
  20. 2023Unified Open-Vocabulary Dense Visual Predictioncited by 4
  21. 2023WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledgecited by 4
  22. 2024In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Modelcited by 4
  23. 2024TP-GMOT: Tracking Generic Multiple Object by Textual Prompt with Motion-Appearance Cost (MAC) SORTcited by 2
  24. 2024FOR: Finetuning for Object Level Open Vocabulary Image Retrievalcited by 2
  25. 2022DetailCLIP: Injecting Image Details into CLIP's Feature Spacecited by 1
  26. 2024Enhanced Kalman with Adaptive Appearance Motion SORT for Grounded Generic Multiple Object Trackingcited by 1
  27. 2025SORCE: Small Object Retrieval in Complex Environmentscited by 1
  28. 2022EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answeringcited by 0
  29. 2026ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Groundingcited by 0