# Interpretable Visual Understanding with Cognitive Attention Network

arXiv: https://arxiv.org/abs/2108.02924 · HTML: https://mlsanity.com/paper/2108.02924/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 95 references that are in the map.
- First posted on arXiv 6 Aug 2021.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [UNITER: UNiversal Image-TExt Representation Learning](https://mlsanity.com/paper/1909.11740/) (2019)
- [ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph](https://mlsanity.com/paper/2006.16934/) (2020)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [Recurrent Neural Network Regularization](https://mlsanity.com/paper/1409.2329/) (2014)
- [FVQA: Fact-based Visual Question Answering](https://mlsanity.com/paper/1606.05433/) (2016)
- [MUTAN: Multimodal Tucker Fusion for Visual Question Answering](https://mlsanity.com/paper/1705.06676/) (2017)

## Built on by (most cited first, 12 of 12)

- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [FairAIED: Navigating Fairness, Bias, and Ethics in Educational AI Applications](https://mlsanity.com/paper/2407.18745/) (2024)
- [Incremental Mining of Frequent Serial Episodes Considering Multiple Occurrences](https://mlsanity.com/paper/2201.11650/) (2022)
- [Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applications](https://mlsanity.com/paper/2510.21887/) (2025)
- [AMCR: A Framework for Assessing and Mitigating Copyright Risks in Generative Models](https://mlsanity.com/paper/2509.00641/) (2025)
- [Towards Transparent AI: A Survey on Explainable Language Models](https://mlsanity.com/paper/2509.21631/) (2025)
- [The Internet of Responsibilities-Connecting Human Responsibilities using Big Data and Blockchain](https://mlsanity.com/paper/2312.04729/) (2023)
- [Uncertain Boundaries: Multidisciplinary Approaches to Copyright Issues in Generative AI](https://mlsanity.com/paper/2404.08221/) (2024)
- [Fairness-Aware Graph Representation Learning with Limited Demographic Information](https://mlsanity.com/paper/2511.13540/) (2025)
- [Attention Mechanism based Cognition-level Scene Understanding](https://mlsanity.com/paper/2204.08027/) (2022)
- [GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMs](https://mlsanity.com/paper/2511.13007/) (2025)

## Abstract (arXiv)

While image understanding on recognition-level has achieved remarkable advancements, reliable visual scene understanding requires comprehensive image understanding on recognition-level but also cognition-level, which calls for exploiting the multi-source information as well as learning different levels of understanding and extensive commonsense knowledge. In this paper, we propose a novel Cognitive Attention Network (CAN) for visual commonsense reasoning to achieve interpretable visual understanding. Specifically, we first introduce an image-text fusion module to fuse information from images and text collectively. Second, a novel inference module is designed to encode commonsense among image, query and response. Extensive experiments on large-scale Visual Commonsense Reasoning (VCR) benchmark dataset demonstrate the effectiveness of our approach. The implementation is publicly available at https://github.com/tanjatang/CAN