# KSAT: Knowledge-infused Self Attention Transformer -- Integrating Multiple Domain-Specific Contexts

arXiv: https://arxiv.org/abs/2210.04307 · HTML: https://mlsanity.com/paper/2210.04307/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 9 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Semantics of the Black-Box: Can knowledge graphs help make deep learning systems more interpretable and explainable?](https://mlsanity.com/paper/2010.08660/) (2020)
- [Knowledge-intensive Language Understanding for Explainable AI](https://mlsanity.com/paper/2108.01174/) (2021)
- [Learning to Automate Follow-up Question Generation using Process Knowledge for Depression Triage on Reddit Posts](https://mlsanity.com/paper/2205.13884/) (2022)
- [Process Knowledge-Infused AI: Towards User-level Explainability, Interpretability, and Safety](https://mlsanity.com/paper/2206.13349/) (2022)
- ["Is depression related to cannabis?": A knowledge-infused model for Entity and Relation Extraction with Limited Supervision](https://mlsanity.com/paper/2102.01222/) (2021)
- ["Who can help me?": Knowledge Infused Matching of Support Seekers and Support Providers during COVID-19 on Reddit](https://mlsanity.com/paper/2105.06398/) (2021)
- [Knowledge Infused Policy Gradients for Adaptive Pandemic Control](https://mlsanity.com/paper/2102.06245/) (2021)
- [NLP is Not enough -- Contextualization of User Input in Chatbots](https://mlsanity.com/paper/2105.06511/) (2021)
- [Knowledge Infused Policy Gradients with Upper Confidence Bound for Relational Bandits](https://mlsanity.com/paper/2106.13895/) (2021)
- [COVID-19 in Spain and India: Comparing Policy Implications by Analyzing Epidemiological and Social Media Data](https://mlsanity.com/paper/2010.14628/) (2020)
- [eDarkTrends: Harnessing Social Media Trends in Substance use disorders for Opioid Listings on Cryptomarket](https://mlsanity.com/paper/2103.15764/) (2021)

## Built on by (most cited first, 5 of 5)

- [ProKnow: Process Knowledge for Safety Constrained and Explainable Question Generation for Mental Health Diagnostic Assistance](https://mlsanity.com/paper/2305.08010/) (2023)
- [Process Knowledge-infused Learning for Clinician-friendly Explanations](https://mlsanity.com/paper/2306.09824/) (2023)
- [Knowledge-Infused Self Attention Transformers](https://mlsanity.com/paper/2306.13501/) (2023)
- [IERL: Interpretable Ensemble Representation Learning -- Combining CrowdSourced Knowledge and Distributed Semantic Representations](https://mlsanity.com/paper/2306.13865/) (2023)
- [L3 Ensembles: Lifelong Learning Approach for Ensemble of Foundational Language Models](https://mlsanity.com/paper/2311.06493/) (2023)

## Abstract (arXiv)

Domain-specific language understanding requires integrating multiple pieces of relevant contextual information. For example, we see both suicide and depression-related behavior (multiple contexts) in the text ``I have a gun and feel pretty bad about my life, and it wouldn't be the worst thing if I didn't wake up tomorrow''. Domain specificity in self-attention architectures is handled by fine-tuning on excerpts from relevant domain specific resources (datasets and external knowledge - medical textbook chapters on mental health diagnosis related to suicide and depression). We propose a modified self-attention architecture Knowledge-infused Self Attention Transformer (KSAT) that achieves the integration of multiple domain-specific contexts through the use of external knowledge sources. KSAT introduces knowledge-guided biases in dedicated self-attention layers for each knowledge source to accomplish this. In addition, KSAT provides mechanics for controlling the trade-off between learning from data and learning from knowledge. Our quantitative and qualitative evaluations show that (1) the KSAT architecture provides novel human-understandable ways to precisely measure and visualize the contributions of the infused domain contexts, and (2) KSAT performs competitively with other knowledge-infused baselines and significantly outperforms baselines that use fine-tuning for domain-specific tasks.