# Research on Named Entity Recognition in Improved transformer with R-Drop structure

arXiv: https://arxiv.org/abs/2306.08315 · HTML: https://mlsanity.com/paper/2306.08315/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Template-Based Named Entity Recognition Using BART](https://mlsanity.com/paper/2106.01760/) (2021)
- [Neural Machine Translation: A Review and Survey](https://mlsanity.com/paper/1912.02047/) (2019)
- [TENER: Adapting Transformer Encoder for Named Entity Recognition](https://mlsanity.com/paper/1911.04474/) (2019)
- [FLAT: Chinese NER Using Flat-Lattice Transformer](https://mlsanity.com/paper/2004.11795/) (2020)
- [Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning](https://mlsanity.com/paper/1906.01378/) (2019)
- [Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training](https://mlsanity.com/paper/2109.05003/) (2021)
- [Simplify the Usage of Lexicon in Chinese NER](https://mlsanity.com/paper/1908.05969/) (2019)
- [Robust Named Entity Recognition with Truecasing Pretraining](https://mlsanity.com/paper/1912.07095/) (2019)
- [MECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition](https://mlsanity.com/paper/2107.05418/) (2021)
- [SLK-NER: Exploiting Second-order Lexicon Knowledge for Chinese NER](https://mlsanity.com/paper/2007.08416/) (2020)
- [Semi-Supervised Disentangled Framework for Transferable Named Entity Recognition](https://mlsanity.com/paper/2012.11805/) (2020)
- [Private Information Retrieval with Side Information](https://mlsanity.com/paper/1709.00112/) (2017)

## Abstract (arXiv)

To enhance the generalization ability of the model and improve the effectiveness of the transformer for named entity recognition tasks, the XLNet-Transformer-R model is proposed in this paper. The XLNet pre-trained model and the Transformer encoder with relative positional encodings are combined to enhance the model's ability to process long text and learn contextual information to improve robustness. To prevent overfitting, the R-Drop structure is used to improve the generalization capability and enhance the accuracy of the model in named entity recognition tasks. The model in this paper performs ablation experiments on the MSRA dataset and comparison experiments with other models on four datasets with excellent performance, demonstrating the strategic effectiveness of the XLNet-Transformer-R model.