# Combining Commonsense Reasoning and Knowledge Acquisition to Guide Deep Learning in Robotics

arXiv: https://arxiv.org/abs/2201.10266 · HTML: https://mlsanity.com/paper/2201.10266/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 26 of its 75 references that are in the map.
- First posted on arXiv 25 Jan 2022.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Neural Probabilistic Logic Programming in DeepProbLog](https://mlsanity.com/paper/1907.08194/) (2019)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models](https://mlsanity.com/paper/1708.08296/) (2017)
- [Neural-Symbolic Learning and Reasoning: A Survey and Interpretation](https://mlsanity.com/paper/1711.03902/) (2017)
- [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018)
- [Label-Free Supervision of Neural Networks with Physics and Domain Knowledge](https://mlsanity.com/paper/1609.05566/) (2016)
- [Learning Physical Intuition of Block Towers by Example](https://mlsanity.com/paper/1603.01312/) (2016)
- [Learning Visual Predictive Models of Physics for Playing Billiards](https://mlsanity.com/paper/1511.07404/) (2015)
- [Neural Programmer: Inducing Latent Programs with Gradient Descent](https://mlsanity.com/paper/1511.04834/) (2015)
- [The ILASP system for Inductive Learning of Answer Set Programs](https://mlsanity.com/paper/2005.00904/) (2020)
- ["What happens if..." Learning to Predict the Effect of Forces in Images](https://mlsanity.com/paper/1603.05600/) (2016)
- [REBA: A Refinement-Based Architecture for Knowledge Representation and Reasoning in Robotics](https://mlsanity.com/paper/1508.03891/) (2015)
- [Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions](https://mlsanity.com/paper/1809.03707/) (2018)
- [A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understanding](https://mlsanity.com/paper/1605.01138/) (2016)
- [Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces](https://mlsanity.com/paper/1904.06827/) (2019)
- [Metric Learning for Generalizing Spatial Relations to New Objects](https://mlsanity.com/paper/1703.01946/) (2017)
- [Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1707.05720/) (2017)
- [Optimization Beyond the Convolution: Generalizing Spatial Relations with End-to-End Metric Learning](https://mlsanity.com/paper/1707.00893/) (2017)
- [Combining Deep Learning and Qualitative Spatial Reasoning to Learn Complex Structures from Sparse Examples with Noise](https://mlsanity.com/paper/1811.11064/) (2018)
- [Axiom Learning and Belief Tracing for Transparent Decision Making in Robotics](https://mlsanity.com/paper/2010.10645/) (2020)
- [Visual Stability Prediction and Its Application to Manipulation](https://mlsanity.com/paper/1609.04861/) (2016)
- [Learning Deep Generative Spatial Models for Mobile Robots](https://mlsanity.com/paper/1610.02627/) (2016)

## Abstract (arXiv)

Algorithms based on deep network models are being used for many pattern recognition and decision-making tasks in robotics and AI. Training these models requires a large labeled dataset and considerable computational resources, which are not readily available in many domains. Also, it is difficult to explore the internal representations and reasoning mechanisms of these models. As a step towards addressing the underlying knowledge representation, reasoning, and learning challenges, the architecture described in this paper draws inspiration from research in cognitive systems. As a motivating example, we consider an assistive robot trying to reduce clutter in any given scene by reasoning about the occlusion of objects and stability of object configurations in an image of the scene. In this context, our architecture incrementally learns and revises a grounding of the spatial relations between objects and uses this grounding to extract spatial information from input images. Non-monotonic logical reasoning with this information and incomplete commonsense domain knowledge is used to make decisions about stability and occlusion. For images that cannot be processed by such reasoning, regions relevant to the tasks at hand are automatically identified and used to train deep network models to make the desired decisions. Image regions used to train the deep networks are also used to incrementally acquire previously unknown state constraints that are merged with the existing knowledge for subsequent reasoning. Experimental evaluation performed using simulated and real-world images indicates that in comparison with baselines based just on deep networks, our architecture improves reliability of decision making and reduces the effort involved in training data-driven deep network models.