mlsanityData to 14 Oct 2026

MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding

arXiv 2104.12763 ↗First posted 26 Apr 2021cs.CV cs.CL cs.LGcited by 819

Not on the map. Why?

Cited by 164 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multi-modal reasoning systems rely on a pre-trained object detector to extract regions of interest from the image. However, this crucial module is typically used as a black box, trained independently of the downstream task and on a fixed vocabulary of objects and attributes. This makes it challenging for such systems to capture the long tail of visual concepts expressed in free form text. In this paper we propose MDETR, an end-to-end modulated detector that detects objects in an image conditioned on a raw text query, like a caption or a question. We use a transformer-based architecture to reason jointly over text and image by fusing the two modalities at an early stage of the model. We pre-train the network on 1.3M text-image pairs, mined from pre-existing multi-modal datasets having explicit alignment between phrases in text and objects in the image. We then fine-tune on several downstream tasks such as phrase grounding, referring expression comprehension and segmentation, achieving state-of-the-art results on popular benchmarks. We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting. We show that our pre-training approach provides a way to handle the long tail of object categories which have very few labelled instances. Our approach can be easily extended for visual question answering, achieving competitive performance on GQA and CLEVR. The code and models are available at https://github.com/ashkamath/mdetr.

Key facts. Cited by 819 papers in the map (data through 14 Oct 2026). Builds on 61 of its 79 references that are in the map. First posted on arXiv 26 Apr 2021.

Builds on

61 of 79 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  4. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  9. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  14. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  15. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  16. 2016Modeling Context in Referring Expressionscited by 1,063
  17. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,020
  18. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  19. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  20. 2017Focal Loss for Dense Object Detectioncited by 752
  21. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  22. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  23. 2017A simple neural network module for relational reasoningcited by 346
  24. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  25. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  26. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  27. 2018Bilinear Attention Networkscited by 280
  28. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  29. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  30. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  31. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195
  32. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  33. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  34. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  35. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  36. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  37. 2019Learning by Abstraction: The Neural State Machinecited by 113
  38. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 153; 2023: 234; 2024: 165; 2025: 121; 2026: 114.

819 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  3. 2021Grounded Language-Image Pre-trainingcited by 1,156
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  6. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  9. 2022Language-driven Semantic Segmentationcited by 717
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  11. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
38 more
  1. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  3. 2021Image Segmentation Using Text and Image Promptscited by 512
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  5. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  6. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  7. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  8. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  9. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  10. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  11. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  12. 2022Visual Language Maps for Robot Navigationcited by 365
  13. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  14. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  15. 2022Visual Classification via Description from Large Language Modelscited by 336
  16. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  17. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  18. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  19. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  20. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  21. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  22. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  23. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  24. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  25. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  26. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  27. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  28. 2023A Cookbook of Self-Supervised Learningcited by 250
  29. 2021Conditional Object-Centric Learning from Videocited by 248
  30. 2022Transformers in Medical Imaging: A Surveycited by 245
  31. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  32. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  33. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  34. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  35. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  36. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  37. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  38. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192

50 most cited of 819 All 819 (JSON)