# DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling

arXiv: https://arxiv.org/abs/2109.08818 · HTML: https://mlsanity.com/paper/2109.08818/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 52 references that are in the map.
- First posted on arXiv 18 Sep 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [Semi-supervised sequence tagging with bidirectional language models](https://mlsanity.com/paper/1705.00108/) (2017)
- [A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1907.00390/) (2019)
- [Chinese NER Using Lattice LSTM](https://mlsanity.com/paper/1805.02023/) (2018)
- [TENER: Adapting Transformer Encoder for Named Entity Recognition](https://mlsanity.com/paper/1911.04474/) (2019)
- [FLAT: Chinese NER Using Flat-Lattice Transformer](https://mlsanity.com/paper/2004.11795/) (2020)
- [Glyce: Glyph-vectors for Chinese Character Representations](https://mlsanity.com/paper/1901.10125/) (2019)
- [Fast and Accurate Entity Recognition with Iterated Dilated Convolutions](https://mlsanity.com/paper/1702.02098/) (2017)
- [Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks](https://mlsanity.com/paper/1703.06345/) (2017)
- [Simplify the Usage of Lexicon in Chinese NER](https://mlsanity.com/paper/1908.05969/) (2019)
- [GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognition](https://mlsanity.com/paper/1907.05611/) (2019)
- [F-Score Driven Max Margin Neural Network for Named Entity Recognition in Chinese Social Media](https://mlsanity.com/paper/1611.04234/) (2016)
- [Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning](https://mlsanity.com/paper/1903.04190/) (2019)
- [Neural Word Segmentation with Rich Pretraining](https://mlsanity.com/paper/1704.08960/) (2017)
- [State-of-the-art Chinese Word Segmentation with Bi-LSTMs](https://mlsanity.com/paper/1808.06511/) (2018)

## Built on by (most cited first, 4 of 4)

- [TURNER: The Uncertainty-based Retrieval Framework for Chinese NER](https://mlsanity.com/paper/2202.09022/) (2022)
- [MedDec: A Dataset for Extracting Medical Decisions from Discharge Summaries](https://mlsanity.com/paper/2408.12980/) (2024)
- [Dynamic Multi-View Fusion Mechanism For Chinese Relation Extraction](https://mlsanity.com/paper/2303.05082/) (2023)
- [HREB-CRF: Hierarchical Reduced-bias EMA for Chinese Named Entity Recognition](https://mlsanity.com/paper/2503.01217/) (2025)

## Abstract (arXiv)

Incorporating lexical knowledge into deep learning models has been proved to be very effective for sequence labeling tasks. However, previous works commonly have difficulty dealing with large-scale dynamic lexicons which often cause excessive matching noise and problems of frequent updates. In this paper, we propose DyLex, a plug-in lexicon incorporation approach for BERT based sequence labeling tasks. Instead of leveraging embeddings of words in the lexicon as in conventional methods, we adopt word-agnostic tag embeddings to avoid re-training the representation while updating the lexicon. Moreover, we employ an effective supervised lexical knowledge denoising method to smooth out matching noise. Finally, we introduce a col-wise attention based knowledge fusion mechanism to guarantee the pluggability of the proposed framework. Experiments on ten datasets of three tasks show that the proposed framework achieves new SOTA, even with very large scale lexicons.