# Common Sense Knowledge Learning for Open Vocabulary Neural Reasoning: A First View into Chronic Disease Literature

arXiv: https://arxiv.org/abs/2111.13781 · HTML: https://mlsanity.com/paper/2111.13781/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 63 references that are in the map.
- First posted on arXiv 27 Nov 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [An Attentive Survey of Attention Models](https://mlsanity.com/paper/1904.02874/) (2019)
- [Differentiable Reasoning over a Virtual Knowledge Base](https://mlsanity.com/paper/2002.10640/) (2020)
- [Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Review](https://mlsanity.com/paper/1908.05780/) (2019)
- [Robust Transfer Learning with Pretrained Language Models through Adapters](https://mlsanity.com/paper/2108.02340/) (2021)
- [Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems](https://mlsanity.com/paper/2109.15142/) (2021)
- [Incremental Knowledge Base Construction Using DeepDive](https://mlsanity.com/paper/1502.00731/) (2015)
- [WiRe57 : A Fine-Grained Benchmark for Open Information Extraction](https://mlsanity.com/paper/1809.08962/) (2018)
- [Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT](https://mlsanity.com/paper/2109.04810/) (2021)
- [Medical Entity Disambiguation Using Graph Neural Networks](https://mlsanity.com/paper/2104.01488/) (2021)
- [Unsupervised Sentence Representations as Word Information Series: Revisiting TF--IDF](https://mlsanity.com/paper/1710.06524/) (2017)
- [Graphene: A Context-Preserving Open Information Extraction System](https://mlsanity.com/paper/1808.09463/) (2018)

## Abstract (arXiv)

In this paper, we address reasoning tasks from open vocabulary Knowledge Bases (openKBs) using state-of-the-art Neural Language Models (NLMs) with applications in scientific literature. For this purpose, self-attention based NLMs are trained using a common sense KB as a source task. The NLMs are then tested on a target KB for open vocabulary reasoning tasks involving scientific knowledge related to the most prevalent chronic diseases (also known as non-communicable diseases, NCDs). Our results identified NLMs that performed consistently and with significance in knowledge inference for both source and target tasks. Furthermore, in our analysis by inspection we discussed the semantic regularities and reasoning capabilities learned by the models, while showing a first insight into the potential benefits of our approach to aid NCD research.