# Deep Neural Networks for Visual Reasoning

arXiv: https://arxiv.org/abs/2209.11990 · HTML: https://mlsanity.com/paper/2209.11990/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 232 references that are in the map.
- First posted on arXiv 24 Sep 2022.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [What Can Neural Networks Reason About?](https://mlsanity.com/paper/1905.13211/) (2019)
- [The Consciousness Prior](https://mlsanity.com/paper/1709.08568/) (2017)
- [Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation](https://mlsanity.com/paper/2007.01951/) (2020)
- [Simple Baseline for Visual Question Answering](https://mlsanity.com/paper/1512.02167/) (2015)
- [Answer Questions with Right Image Regions: A Visual Attention Regularization Approach](https://mlsanity.com/paper/2102.01916/) (2021)
- [Learning from Lexical Perturbations for Consistent Visual Question Answering](https://mlsanity.com/paper/2011.13406/) (2020)
- [A Comparative Study of Word Embeddings for Reading Comprehension](https://mlsanity.com/paper/1703.00993/) (2017)
- [Regularizing Attention Networks for Anomaly Detection in Visual Question Answering](https://mlsanity.com/paper/2009.10054/) (2020)
- [Visual Question Answering with Prior Class Semantics](https://mlsanity.com/paper/2005.01239/) (2020)
- [Learning Deep Matrix Representations](https://mlsanity.com/paper/1703.01454/) (2017)

## Abstract (arXiv)

Visual perception and language understanding are - fundamental components of human intelligence, enabling them to understand and reason about objects and their interactions. It is crucial for machines to have this capacity to reason using these two modalities to invent new robot-human collaborative systems. Recent advances in deep learning have built separate sophisticated representations of both visual scenes and languages. However, understanding the associations between the two modalities in a shared context for multimodal reasoning remains a challenge. Focusing on language and vision modalities, this thesis advances the understanding of how to exploit and use pivotal aspects of vision-and-language tasks with neural networks to support reasoning. We derive these understandings from a series of works, making a two-fold contribution: (i) effective mechanisms for content selection and construction of temporal relations from dynamic visual scenes in response to a linguistic query and preparing adequate knowledge for the reasoning process (ii) new frameworks to perform reasoning with neural networks by exploiting visual-linguistic associations, deduced either directly from data or guided by external priors.