# Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

arXiv: https://arxiv.org/abs/2007.00849 · HTML: https://mlsanity.com/paper/2007.00849/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2020.

## Built on by (most cited first, 30 of 37)

- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [End-to-End Training of Neural Retrievers for Open-Domain Question Answering](https://mlsanity.com/paper/2101.00408/) (2021)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Fine-tuning Image Transformers using Learnable Memory](https://mlsanity.com/paper/2203.15243/) (2022)
- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [Relational World Knowledge Representation in Contextual Language Models: A Review](https://mlsanity.com/paper/2104.05837/) (2021)
- [Knowledge Base Question Answering by Case-based Reasoning over Subgraphs](https://mlsanity.com/paper/2202.10610/) (2022)
- [ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers](https://mlsanity.com/paper/2110.06884/) (2021)
- [ASER: Towards Large-scale Commonsense Knowledge Acquisition via Higher-order Selectional Preference over Eventualities](https://mlsanity.com/paper/2104.02137/) (2021)
- [Emptying the Ocean with a Spoon: Should We Edit Models?](https://mlsanity.com/paper/2310.11958/) (2023)
- [Highly Parallel Autoregressive Entity Linking with Discriminative Correction](https://mlsanity.com/paper/2109.03792/) (2021)
- [Cross-Modal Retrieval Augmentation for Multi-Modal Classification](https://mlsanity.com/paper/2104.08108/) (2021)
- [Reasoning Over Virtual Knowledge Bases With Open Predicate Relations](https://mlsanity.com/paper/2102.07043/) (2021)
- [Can Language Models Act as Knowledge Bases at Scale?](https://mlsanity.com/paper/2402.14273/) (2024)
- [Memorization in deep learning: A survey](https://mlsanity.com/paper/2406.03880/) (2024)
- [Iterative Hierarchical Attention for Answering Complex Questions over Long Documents](https://mlsanity.com/paper/2106.00200/) (2021)
- [Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems](https://mlsanity.com/paper/2203.10610/) (2022)
- [Neuro-Symbolic AI: Explainability, Challenges, and Future Trends](https://mlsanity.com/paper/2411.04383/) (2024)
- [Relation-Guided Pre-Training for Open-Domain Question Answering](https://mlsanity.com/paper/2109.10346/) (2021)
- [Unlocking the Potential of Generative AI through Neuro-Symbolic Architectures: Benefits and Limitations](https://mlsanity.com/paper/2502.11269/) (2025)
- [Single Sequence Prediction over Reasoning Graphs for Multi-hop QA](https://mlsanity.com/paper/2307.00335/) (2023)
- [PInKS: Preconditioned Commonsense Inference with Minimal Supervision](https://mlsanity.com/paper/2206.07920/) (2022)
- [Trusted Source Alignment in Large Language Models](https://mlsanity.com/paper/2311.06697/) (2023)
- [Does Entity Abstraction Help Generative Transformers Reason?](https://mlsanity.com/paper/2201.01787/) (2022)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)

## Abstract (arXiv)

Massive language models are the core of modern NLP modeling and have been shown to encode impressive amounts of commonsense and factual information. However, that knowledge exists only within the latent parameters of the model, inaccessible to inspection and interpretation, and even worse, factual information memorized from the training corpora is likely to become stale as the world changes. Knowledge stored as parameters will also inevitably exhibit all of the biases inherent in the source materials. To address these problems, we develop a neural language model that includes an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge. We show that this model dramatically improves performance on two knowledge-intensive question-answering tasks. More interestingly, the model can be updated without re-training by manipulating its symbolic representations. In particular this model allows us to add new facts and overwrite existing ones in ways that are not possible for earlier models.