mlsanityData to 7 Oct 2026

In Defense of Grid Features for Visual Question Answering

arXiv 2001.03615 ↗First posted 10 Jan 2020cs.CVcited by 155

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Popularized as 'bottom-up' attention, bounding box (or region) based visual features have recently surpassed vanilla grid-based convolutional features as the de facto standard for vision and language tasks like visual question answering (VQA). However, it is not clear whether the advantages of regions (e.g. better localization) are the key reasons for the success of bottom-up attention. In this paper, we revisit grid features for VQA, and find they can work surprisingly well - running more than an order of magnitude faster with the same accuracy (e.g. if pre-trained in a similar fashion). Through extensive experiments, we verify that this observation holds true across different VQA models (reporting a state-of-the-art accuracy on VQA 2.0 test-std, 72.71), datasets, and generalizes well to other tasks like image captioning. As grid features make the model design and training process much simpler, this enables us to train them end-to-end and also use a more flexible network design. We learn VQA models end-to-end, from pixels directly to answers, and show that strong performance is achievable without using any region annotations in pre-training. We hope our findings help further improve the scientific understanding and the practical application of VQA. Code and features will be made available.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 47; 2022: 48; 2023: 35; 2024: 14; 2025: 7; 2026: 4.

155 papers.

Sort:

By area:

  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  3. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  4. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  5. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  6. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  7. 2021Mesh Graphormercited by 264
  8. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  9. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  10. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  11. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  12. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
38 more
  1. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  3. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  4. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  5. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  6. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  7. 2021WebQA: Multihop and Multimodal QAcited by 109
  8. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  9. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  10. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  11. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  12. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  13. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84
  14. 2022A Survey on Long-Tailed Visual Recognitioncited by 75
  15. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  16. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  17. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  18. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
  19. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  20. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  21. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  22. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 45
  23. 2024Context-Guided Spatio-Temporal Video Groundingcited by 45
  24. 2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41
  25. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  26. 2022Gender and Racial Bias in Visual Question Answering Datasetscited by 39
  27. 2021Unified Graph Structured Models for Video Understandingcited by 38
  28. 2022From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Datacited by 38
  29. 2022End-to-End Transformer Based Model for Image Captioningcited by 37
  30. 2022Quantifying Societal Bias Amplification in Image Captioningcited by 37
  31. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  32. 2021Dual-Key Multimodal Backdoors for Visual Question Answeringcited by 35
  33. 2022Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Taskcited by 35
  34. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  35. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  36. 2023Q2ATransformer: Improving Medical VQA via an Answer Querying Decodercited by 31
  37. 2023Towards Local Visual Modeling for Image Captioningcited by 26
  38. 2021Coarse-to-Fine Reasoning for Visual Question Answeringcited by 25

50 most cited of 155 All 155 (JSON)