# Learning to Count Objects in Natural Images for Visual Question Answering

arXiv: https://arxiv.org/abs/1802.05766 · HTML: https://mlsanity.com/paper/1802.05766/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2018.

## Built on by (most cited first, 30 of 40)

- [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data](https://mlsanity.com/paper/2205.03147/) (2022)
- [MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2203.09138/) (2022)
- [Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation](https://mlsanity.com/paper/2103.05271/) (2021)
- [Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models](https://mlsanity.com/paper/2312.02219/) (2023)
- [Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics](https://mlsanity.com/paper/2202.00259/) (2022)
- [On Modality Bias Recognition and Reduction](https://mlsanity.com/paper/2202.12690/) (2022)
- [AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss](https://mlsanity.com/paper/2105.01993/) (2021)
- [BGT-Net: Bidirectional GRU Transformer Network for Scene Graph Generation](https://mlsanity.com/paper/2109.05346/) (2021)
- [Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem](https://mlsanity.com/paper/2207.11850/) (2022)
- [VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13116/) (2021)
- [On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applications](https://mlsanity.com/paper/2312.17016/) (2023)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [Guiding Visual Question Answering with Attention Priors](https://mlsanity.com/paper/2205.12616/) (2022)
- [Emergent Symbol-like Number Variables in Artificial Neural Networks](https://mlsanity.com/paper/2501.06141/) (2025)
- [Semantic-aware Modular Capsule Routing for Visual Question Answering](https://mlsanity.com/paper/2207.10404/) (2022)
- [Visual Enumeration Remains Challenging for Multimodal Generative AI](https://mlsanity.com/paper/2402.03328/) (2024)
- [Bilateral Cross-Modality Graph Matching Attention for Feature Fusion in Visual Question Answering](https://mlsanity.com/paper/2112.07270/) (2021)
- [Detection-Driven Object Count Optimization for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2408.11721/) (2024)
- [QUOTA: Quantifying Objects with Text-to-Image Models for Any Domain](https://mlsanity.com/paper/2411.19534/) (2024)
- [UNICBench: UNIfied Counting Benchmark for MLLM](https://mlsanity.com/paper/2603.00595/) (2026)
- [HySTER: A Hybrid Spatio-Temporal Event Reasoner](https://mlsanity.com/paper/2101.06644/) (2021)
- [Visual Question Answering based on Local-Scene-Aware Referring Expression Generation](https://mlsanity.com/paper/2101.08978/) (2021)
- [MGA-VQA: Multi-Granularity Alignment for Visual Question Answering](https://mlsanity.com/paper/2201.10656/) (2022)
- [Object Attribute Matters in Visual Question Answering](https://mlsanity.com/paper/2401.09442/) (2023)
- [Quantity Matters: Towards Assessing and Mitigating Number Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2403.01373/) (2024)

## Abstract (arXiv)

Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-the-art accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.