# Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution

arXiv: https://arxiv.org/abs/2103.14230 · HTML: https://mlsanity.com/paper/2103.14230/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 78 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [A Theory of Generative ConvNet](https://mlsanity.com/paper/1602.03264/) (2016)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [MarrNet: 3D Shape Reconstruction via 2.5D Sketches](https://mlsanity.com/paper/1711.03129/) (2017)
- [Stratified Rule-Aware Network for Abstract Visual Reasoning](https://mlsanity.com/paper/2002.06838/) (2020)
- [Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense](https://mlsanity.com/paper/1909.01507/) (2019)
- [Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1808.02201/) (2018)
- [Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense](https://mlsanity.com/paper/2004.09044/) (2020)
- [Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics](https://mlsanity.com/paper/1706.04317/) (2017)
- [Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning](https://mlsanity.com/paper/2006.06649/) (2020)
- [Learning Perceptual Inference by Contrasting](https://mlsanity.com/paper/1912.00086/) (2019)
- [Learning to Make Analogies by Contrasting Abstract Relational Structure](https://mlsanity.com/paper/1902.00120/) (2019)
- [Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation](https://mlsanity.com/paper/1810.13049/) (2018)
- [Visual Concept-Metaconcept Learning](https://mlsanity.com/paper/2002.01464/) (2020)
- [Abstract Diagrammatic Reasoning with Multiplex Graph Networks](https://mlsanity.com/paper/2006.11197/) (2020)
- [Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model](https://mlsanity.com/paper/1812.10907/) (2018)
- [Abstract Reasoning with Distracting Features](https://mlsanity.com/paper/1912.00569/) (2019)
- [Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations](https://mlsanity.com/paper/1811.04784/) (2018)
- [Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patterns](https://mlsanity.com/paper/1909.11975/) (2019)
- [Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning](https://mlsanity.com/paper/2004.12193/) (2020)
- [MetaStyle: Three-Way Trade-Off Among Speed, Flexibility, and Quality in Neural Style Transfer](https://mlsanity.com/paper/1812.05233/) (2018)
- [HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving](https://mlsanity.com/paper/2102.11344/) (2021)
- [Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning](https://mlsanity.com/paper/1911.11185/) (2019)

## Built on by (most cited first, 30 of 43)

- [EgoTaskQA: Understanding Human Tasks in Egocentric Videos](https://mlsanity.com/paper/2210.03929/) (2022)
- [A Neuro-vector-symbolic Architecture for Solving Raven's Progressive Matrices](https://mlsanity.com/paper/2203.04571/) (2022)
- [Neuro-Symbolic AI in 2024: A Systematic Review](https://mlsanity.com/paper/2501.05435/) (2025)
- [Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices](https://mlsanity.com/paper/2201.12382/) (2022)
- [A Review of Emerging Research Directions in Abstract Visual Reasoning](https://mlsanity.com/paper/2202.10284/) (2022)
- [Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning](https://mlsanity.com/paper/2111.12990/) (2021)
- [In-Context Analogical Reasoning with Pre-Trained Language Models](https://mlsanity.com/paper/2305.17626/) (2023)
- [GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs](https://mlsanity.com/paper/2311.04901/) (2023)
- [MEWL: Few-shot multimodal word learning with referential uncertainty](https://mlsanity.com/paper/2306.00503/) (2023)
- [Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images](https://mlsanity.com/paper/2110.00519/) (2021)
- [Abstract Visual Reasoning: An Algebraic Approach for Solving Raven's Progressive Matrices](https://mlsanity.com/paper/2303.11730/) (2023)
- [Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in Vector-symbolic Architectures](https://mlsanity.com/paper/2401.16024/) (2024)
- [Towards Learning Abductive Reasoning using VSA Distributed Representations](https://mlsanity.com/paper/2406.19121/) (2024)
- [Interactive Visual Reasoning under Uncertainty](https://mlsanity.com/paper/2206.09203/) (2022)
- [Active Reasoning in an Open-World Environment](https://mlsanity.com/paper/2311.02018/) (2023)
- [Neuro-Symbolic Learning: Principles and Applications in Ophthalmology](https://mlsanity.com/paper/2208.00374/) (2022)
- [Multi-Viewpoint and Multi-Evaluation with Felicitous Inductive Bias Boost Machine Abstract Reasoning Ability](https://mlsanity.com/paper/2210.14914/) (2022)
- [Towards Efficient Neuro-Symbolic AI: From Workload Characterization to Hardware Architecture](https://mlsanity.com/paper/2409.13153/) (2024)
- [Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning](https://mlsanity.com/paper/2412.05586/) (2024)
- [Abstracting Concept-Changing Rules for Solving Raven's Progressive Matrix Problems](https://mlsanity.com/paper/2307.07734/) (2023)
- [Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models](https://mlsanity.com/paper/2311.09064/) (2023)
- [Integration of cognitive tasks into artificial general intelligence test for large models](https://mlsanity.com/paper/2402.02547/) (2024)
- [Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?](https://mlsanity.com/paper/2503.11207/) (2025)
- [DETR++: Taming Your Multi-Scale Detection Transformer](https://mlsanity.com/paper/2206.02977/) (2022)
- [A Cognitively-Inspired Neural Architecture for Visual Abstract Reasoning Using Contrastive Perceptual and Conceptual Processing](https://mlsanity.com/paper/2309.10532/) (2023)
- [Sandra -- A Neuro-Symbolic Reasoner Based On Descriptions And Situations](https://mlsanity.com/paper/2402.00591/) (2024)
- [ViTCN: Vision Transformer Contrastive Network For Reasoning](https://mlsanity.com/paper/2403.09962/) (2024)
- [A-I-RAVEN and I-RAVEN-Mesh: Two New Benchmarks for Abstract Visual Reasoning](https://mlsanity.com/paper/2406.11061/) (2024)
- [Learning Visual Abstract Reasoning through Dual-Stream Networks](https://mlsanity.com/paper/2411.19451/) (2024)
- [On the Complexity of Bayesian Generalization](https://mlsanity.com/paper/2211.11033/) (2022)

## Abstract (arXiv)

Spatial-temporal reasoning is a challenging task in Artificial Intelligence (AI) due to its demanding but unique nature: a theoretic requirement on representing and reasoning based on spatial-temporal knowledge in mind, and an applied requirement on a high-level cognitive system capable of navigating and acting in space and time. Recent works have focused on an abstract reasoning task of this kind -- Raven's Progressive Matrices (RPM). Despite the encouraging progress on RPM that achieves human-level performance in terms of accuracy, modern approaches have neither a treatment of human-like reasoning on generalization, nor a potential to generate answers. To fill in this gap, we propose a neuro-symbolic Probabilistic Abduction and Execution (PrAE) learner; central to the PrAE learner is the process of probabilistic abduction and execution on a probabilistic scene representation, akin to the mental manipulation of objects. Specifically, we disentangle perception and reasoning from a monolithic model. The neural visual perception frontend predicts objects' attributes, later aggregated by a scene inference engine to produce a probabilistic scene representation. In the symbolic logical reasoning backend, the PrAE learner uses the representation to abduce the hidden rules. An answer is predicted by executing the rules on the probabilistic representation. The entire system is trained end-to-end in an analysis-by-synthesis manner without any visual attribute annotations. Extensive experiments demonstrate that the PrAE learner improves cross-configuration generalization and is capable of rendering an answer, in contrast to prior works that merely make a categorical choice from candidates.