# Object-Centric Diagnosis of Visual Reasoning

arXiv: https://arxiv.org/abs/2012.11587 · HTML: https://mlsanity.com/paper/2012.11587/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2020.

## Built on by (most cited first, 16 of 16)

- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [3D Question Answering](https://mlsanity.com/paper/2112.08359/) (2021)
- [From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data](https://mlsanity.com/paper/2205.03147/) (2022)
- [ProTo: Program-Guided Transformer for Program-Guided Tasks](https://mlsanity.com/paper/2110.00804/) (2021)
- [Data Efficient Masked Language Modeling for Vision and Language](https://mlsanity.com/paper/2109.02040/) (2021)
- [GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs](https://mlsanity.com/paper/2311.04901/) (2023)
- [Object-centric Learning with Cyclic Walks between Parts and Whole](https://mlsanity.com/paper/2302.08023/) (2023)
- [VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives](https://mlsanity.com/paper/2206.11212/) (2022)
- [Object-Centric Learning with Slot Mixture Module](https://mlsanity.com/paper/2311.04640/) (2023)
- [Are We Done with Object-Centric Learning?](https://mlsanity.com/paper/2504.07092/) (2025)
- [Object-Centric Representation Learning with Generative Spatial-Temporal Factorization](https://mlsanity.com/paper/2111.05393/) (2021)
- [Cycle Consistency Driven Object Discovery](https://mlsanity.com/paper/2306.02204/) (2023)
- [KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence](https://mlsanity.com/paper/2103.00519/) (2021)
- [Object-Centric Temporal Consistency via Conditional Autoregressive Inductive Biases](https://mlsanity.com/paper/2410.15728/) (2024)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)

## Abstract (arXiv)

When answering questions about an image, it not only needs knowing what -- understanding the fine-grained contents (e.g., objects, relationships) in the image, but also telling why -- reasoning over grounding visual cues to derive the answer for a question. Over the last few years, we have seen significant progress on visual question answering. Though impressive as the accuracy grows, it still lags behind to get knowing whether these models are undertaking grounding visual reasoning or just leveraging spurious correlations in the training data. Recently, a number of works have attempted to answer this question from perspectives such as grounding and robustness. However, most of them are either focusing on the language side or coarsely studying the pixel-level attention maps. In this paper, by leveraging the step-wise object grounding annotations provided in the GQA dataset, we first present a systematical object-centric diagnosis of visual reasoning on grounding and robustness, particularly on the vision side. According to the extensive comparisons across different models, we find that even models with high accuracy are not good at grounding objects precisely, nor robust to visual content perturbations. In contrast, symbolic and modular models have a relatively better grounding and robustness, though at the cost of accuracy. To reconcile these different aspects, we further develop a diagnostic model, namely Graph Reasoning Machine. Our model replaces purely symbolic visual representation with probabilistic scene graph and then applies teacher-forcing training for the visual reasoning module. The designed model improves the performance on all three metrics over the vanilla neural-symbolic model while inheriting the transparency. Further ablation studies suggest that this improvement is mainly due to more accurate image understanding and proper intermediate reasoning supervisions.