# Knowledge-Infused Self Attention Transformers

arXiv: https://arxiv.org/abs/2306.13501 · HTML: https://mlsanity.com/paper/2306.13501/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Knowledge-intensive Language Understanding for Explainable AI](https://mlsanity.com/paper/2108.01174/) (2021)
- [Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts](https://mlsanity.com/paper/2205.13884/) (2022)
- [Neurosymbolic AI -- Why, What, and How](https://mlsanity.com/paper/2305.00813/) (2023)
- [Process Knowledge-Infused AI: Towards User-level Explainability, Interpretability, and Safety](https://mlsanity.com/paper/2206.13349/) (2022)
- [ProKnow: Process Knowledge for Safety Constrained and Explainable Question Generation for Mental Health Diagnostic Assistance](https://mlsanity.com/paper/2305.08010/) (2023)
- [Process Knowledge-infused Learning for Clinician-friendly Explanations](https://mlsanity.com/paper/2306.09824/) (2023)
- ["Is depression related to cannabis?": A knowledge-infused model for Entity and Relation Extraction with Limited Supervision](https://mlsanity.com/paper/2102.01222/) (2021)
- ["Who can help me?": Knowledge Infused Matching of Support Seekers and Support Providers during COVID-19 on Reddit](https://mlsanity.com/paper/2105.06398/) (2021)
- [Knowledge Infused Policy Gradients for Adaptive Pandemic Control](https://mlsanity.com/paper/2102.06245/) (2021)
- [NLP is Not enough -- Contextualization of User Input in Chatbots](https://mlsanity.com/paper/2105.06511/) (2021)
- [KSAT: Knowledge-infused Self Attention Transformer -- Integrating Multiple Domain-Specific Contexts](https://mlsanity.com/paper/2210.04307/) (2022)
- [COVID-19 in Spain and India: Comparing Policy Implications by Analyzing Epidemiological and Social Media Data](https://mlsanity.com/paper/2010.14628/) (2020)
- [Demo Alleviate: Demonstrating Artificial Intelligence Enabled Virtual Assistance for Telehealth: The Mental Health Case](https://mlsanity.com/paper/2304.00025/) (2023)
- [eDarkTrends: Harnessing Social Media Trends in Substance use disorders for Opioid Listings on Cryptomarket](https://mlsanity.com/paper/2103.15764/) (2021)
- [Knowledge Graph Guided Semantic Evaluation of Language Models For User Trust](https://mlsanity.com/paper/2305.04989/) (2023)
- [Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipes](https://mlsanity.com/paper/2306.01805/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [AE-GPT: Using Large Language Models to Extract Adverse Events from Surveillance Reports-A Use Case with Influenza Vaccine Adverse Events](https://mlsanity.com/paper/2309.16150/) (2023)
- [L3 Ensembles: Lifelong Learning Approach for Ensemble of Foundational Language Models](https://mlsanity.com/paper/2311.06493/) (2023)

## Abstract (arXiv)

Transformer-based language models have achieved impressive success in various natural language processing tasks due to their ability to capture complex dependencies and contextual information using self-attention mechanisms. However, they are not without limitations. These limitations include hallucinations, where they produce incorrect outputs with high confidence, and alignment issues, where they generate unhelpful and unsafe outputs for human users. These limitations stem from the absence of implicit and missing context in the data alone. To address this, researchers have explored augmenting these models with external knowledge from knowledge graphs to provide the necessary additional context. However, the ad-hoc nature of existing methods makes it difficult to properly analyze the effects of knowledge infusion on the many moving parts or components of a transformer. This paper introduces a systematic method for infusing knowledge into different components of a transformer-based model. A modular framework is proposed to identify specific components within the transformer architecture, such as the self-attention mechanism, encoder layers, or the input embedding layer, where knowledge infusion can be applied. Additionally, extensive experiments are conducted on the General Language Understanding Evaluation (GLUE) benchmark tasks, and the findings are reported. This systematic approach aims to facilitate more principled approaches to incorporating knowledge into language model architectures.