# Differentiable Reasoning over Long Stories -- Assessing Systematic Generalisation in Neural Models

arXiv: https://arxiv.org/abs/2203.10620 · HTML: https://mlsanity.com/paper/2203.10620/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Learning Explanatory Rules from Noisy Data](https://mlsanity.com/paper/1711.04574/) (2017)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text](https://mlsanity.com/paper/1908.06177/) (2019)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [Learning by Abstraction: The Neural State Machine](https://mlsanity.com/paper/1907.03950/) (2019)
- [Attention-based Graph Neural Network for Semi-supervised Learning](https://mlsanity.com/paper/1803.03735/) (2018)
- [Neural Module Networks for Reasoning over Text](https://mlsanity.com/paper/1912.04971/) (2019)
- [Learning Reasoning Strategies in End-to-End Differentiable Proving](https://mlsanity.com/paper/2007.06477/) (2020)
- [VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation](https://mlsanity.com/paper/1708.04686/) (2017)
- [Towards Training Recurrent Neural Networks for Lifelong Learning](https://mlsanity.com/paper/1811.07017/) (2018)

## Abstract (arXiv)

Contemporary neural networks have achieved a series of developments and successes in many aspects; however, when exposed to data outside the training distribution, they may fail to predict correct answers. In this work, we were concerned about this generalisation issue and thus analysed a broad set of models systematically and robustly over long stories. Related experiments were conducted based on the CLUTRR, which is a diagnostic benchmark suite that can analyse generalisation of natural language understanding (NLU) systems by training over small story graphs and testing on larger ones. In order to handle the multi-relational story graph, we consider two classes of neural models: "E-GNN", the graph-based models that can process graph-structured data and consider the edge attributes simultaneously; and "L-Graph", the sequence-based models which can process linearized version of the graphs. We performed an extensive empirical evaluation, and we found that the modified recurrent neural network yield surprisingly accurate results across every systematic generalisation tasks which outperform the modified graph neural network, while the latter produced more robust models.