# Visual Abductive Reasoning

arXiv: https://arxiv.org/abs/2203.14040 · HTML: https://mlsanity.com/paper/2203.14040/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 49 of its 86 references that are in the map.
- First posted on arXiv 26 Mar 2022.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [End-to-End Dense Video Captioning with Masked Transformer](https://mlsanity.com/paper/1804.00819/) (2018)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [Object Relational Graph with Teacher-Recommended Learning for Video Captioning](https://mlsanity.com/paper/2002.11566/) (2020)
- [Cross-Modal and Hierarchical Modeling of Video and Text](https://mlsanity.com/paper/1810.07212/) (2018)
- [Translating Videos to Natural Language Using Deep Recurrent Neural Networks](https://mlsanity.com/paper/1412.4729/) (2014)

## Built on by (most cited first, 30 of 30)

- [Decoupling Features in Hierarchical Propagation for Video Object Segmentation](https://mlsanity.com/paper/2210.09782/) (2022)
- [TRACE: Temporal Grounding Video LLM via Causal Event Modeling](https://mlsanity.com/paper/2410.05643/) (2024)
- [DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2303.03032/) (2023)
- [Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2211.02219/) (2022)
- [Lana: A Language-Capable Navigator for Instruction Following and Generation](https://mlsanity.com/paper/2303.08409/) (2023)
- [PDPP: Projected Diffusion for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2303.14676/) (2023)
- [Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing](https://mlsanity.com/paper/2210.15889/) (2022)
- [MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning](https://mlsanity.com/paper/2409.17647/) (2024)
- [Iterative Scene Graph Generation with Generative Transformers](https://mlsanity.com/paper/2211.16636/) (2022)
- [FRAG: Frequency Adapting Group for Diffusion Video Editing](https://mlsanity.com/paper/2406.06044/) (2024)
- [A negation detection assessment of GPTs: analysis with the xNot360 dataset](https://mlsanity.com/paper/2306.16638/) (2023)
- [Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events](https://mlsanity.com/paper/2412.05725/) (2024)
- [Active Reasoning in an Open-World Environment](https://mlsanity.com/paper/2311.02018/) (2023)
- [What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning](https://mlsanity.com/paper/2503.21055/) (2025)
- [MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoning](https://mlsanity.com/paper/2501.07227/) (2025)
- [Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMs](https://mlsanity.com/paper/2604.08016/) (2026)
- [Exploring the Potential of Multi-Modal AI for Driving Hazard Prediction](https://mlsanity.com/paper/2310.04671/) (2023)
- [DixitWorld: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay](https://mlsanity.com/paper/2510.10117/) (2025)
- [Visual Transformation Telling](https://mlsanity.com/paper/2305.01928/) (2023)
- [Video Timeline Modeling For News Story Understanding](https://mlsanity.com/paper/2309.13446/) (2023)
- [Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoning](https://mlsanity.com/paper/2506.07811/) (2025)
- [Personality-aware Human-centric Multimodal Reasoning: A New Task, Dataset and Baselines](https://mlsanity.com/paper/2304.02313/) (2023)
- [Learning to Visually Connect Actions and their Effects](https://mlsanity.com/paper/2401.10805/) (2024)
- [Inferring Past Human Actions in Homes with Abductive Reasoning](https://mlsanity.com/paper/2210.13984/) (2022)
- [Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning](https://mlsanity.com/paper/2608.15869/) (2026)
- [EgoVIS@CVPR: What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning](https://mlsanity.com/paper/2506.00101/) (2025)
- [Who Benefits from AI Explanations? Towards Accessible and Interpretable Systems](https://mlsanity.com/paper/2508.10806/) (2025)
- [MSAM: Multi-Semantic Adaptive Mining for Cross-Modal Drone Video-Text Retrieval](https://mlsanity.com/paper/2510.15470/) (2025)
- [IntentQA: Intent Question Answering in Videos by Cognitive Context Reasoning](https://mlsanity.com/paper/2608.23330/) (2026)
- [Think-Verify-Revise: Neuro-Symbolic Visual Reasoning with Vision-Language Models and Dynamic Logic Tensor Networks](https://mlsanity.com/paper/2609.05388/) (2026)

## Abstract (arXiv)

Abductive reasoning seeks the likeliest possible explanation for partial observations. Although abduction is frequently employed in human daily reasoning, it is rarely explored in computer vision literature. In this paper, we propose a new task and dataset, Visual Abductive Reasoning (VAR), for examining abductive reasoning ability of machine intelligence in everyday visual situations. Given an incomplete set of visual events, AI systems are required to not only describe what is observed, but also infer the hypothesis that can best explain the visual premise. Based on our large-scale VAR dataset, we devise a strong baseline model, Reasoner (causal-and-cascaded reasoning Transformer). First, to capture the causal structure of the observations, a contextualized directional position embedding strategy is adopted in the encoder, that yields discriminative representations for the premise and hypothesis. Then, multiple decoders are cascaded to generate and progressively refine the premise and hypothesis sentences. The prediction scores of the sentences are used to guide cross-sentence information flow in the cascaded reasoning procedure. Our VAR benchmarking results show that Reasoner surpasses many famous video-language models, while still being far behind human performance. This work is expected to foster future efforts in the reasoning-beyond-observation paradigm.