# KARL-Trans-NER: Knowledge Aware Representation Learning for Named Entity Recognition using Transformers

arXiv: https://arxiv.org/abs/2111.15436 · HTML: https://mlsanity.com/paper/2111.15436/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 61 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [Deep Entity Matching with Pre-Trained Language Models](https://mlsanity.com/paper/2004.00584/) (2020)
- [Star-Transformer](https://mlsanity.com/paper/1902.09113/) (2019)
- [Automated Concatenation of Embeddings for Structured Prediction](https://mlsanity.com/paper/2010.05006/) (2020)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [Joint Embedding of Words and Labels for Text Classification](https://mlsanity.com/paper/1805.04174/) (2018)
- [TENER: Adapting Transformer Encoder for Named Entity Recognition](https://mlsanity.com/paper/1911.04474/) (2019)
- [CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://mlsanity.com/paper/1909.01441/) (2019)
- [Modeling Relation Paths for Representation Learning of Knowledge Bases](https://mlsanity.com/paper/1506.00379/) (2015)
- [GraphIE: A Graph-Based Framework for Information Extraction](https://mlsanity.com/paper/1810.13083/) (2018)
- [Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks](https://mlsanity.com/paper/1607.01426/) (2016)
- [Hierarchical Contextualized Representation for Named Entity Recognition](https://mlsanity.com/paper/1911.02257/) (2019)
- [CoKE: Contextualized Knowledge Graph Embedding](https://mlsanity.com/paper/1911.02168/) (2019)
- [TransG : A Generative Mixture Model for Knowledge Graph Embedding](https://mlsanity.com/paper/1509.05488/) (2015)
- [Exploring Cross-sentence Contexts for Named Entity Recognition with BERT](https://mlsanity.com/paper/2006.01563/) (2020)
- [Sentence-State LSTM for Text Representation](https://mlsanity.com/paper/1805.02474/) (2018)
- [Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by Averaging Source Word Embeddings](https://mlsanity.com/paper/1804.05262/) (2018)
- [Boosting Named Entity Recognition with Neural Character Embeddings](https://mlsanity.com/paper/1505.05008/) (2015)

## Abstract (arXiv)

The inception of modeling contextual information using models such as BERT, ELMo, and Flair has significantly improved representation learning for words. It has also given SOTA results in almost every NLP task - Machine Translation, Text Summarization and Named Entity Recognition, to name a few. In this work, in addition to using these dominant context-aware representations, we propose a Knowledge Aware Representation Learning (KARL) Network for Named Entity Recognition (NER). We discuss the challenges of using existing methods in incorporating world knowledge for NER and show how our proposed methods could be leveraged to overcome those challenges. KARL is based on a Transformer Encoder that utilizes large knowledge bases represented as fact triplets, converts them to a graph context, and extracts essential entity information residing inside to generate contextualized triplet representation for feature augmentation. Experimental results show that the augmentation done using KARL can considerably boost the performance of our NER system and achieve significantly better results than existing approaches in the literature on three publicly available NER datasets, namely CoNLL 2003, CoNLL++, and OntoNotes v5. We also observe better generalization and application to a real-world setting from KARL on unseen entities.