mlsanityData to 7 Oct 2026

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

arXiv 1707.07998 ↗First posted 25 Jul 2017cs.CVcited by 1,310

Not on the map. Why?

Cited by 142 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

Key facts. Cited by 1,310 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 1; 2021: 363; 2022: 335; 2023: 302; 2024: 161; 2025: 109; 2026: 37.

1,310 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  6. 2023A Survey on Multimodal Large Language Modelscited by 898
  7. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  11. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
38 more
  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  2. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  5. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  9. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  10. 2021InfographicVQAcited by 430
  11. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  12. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  13. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  14. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  15. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  16. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  17. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  18. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  19. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  20. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  21. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  22. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  23. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  24. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  25. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  26. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 244
  27. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  28. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  29. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  30. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  31. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  32. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  33. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  34. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  35. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  36. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  37. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  38. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183

50 most cited of 1,310 All 1,310 (JSON)