# UER: An Open-Source Toolkit for Pre-training Models

arXiv: https://arxiv.org/abs/1909.05658 · HTML: https://mlsanity.com/paper/1909.05658/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2019.

## Built on by (most cited first, 30 of 62)

- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2106.08087/) (2021)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification](https://mlsanity.com/paper/2202.06335/) (2022)
- [BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark](https://mlsanity.com/paper/2302.09432/) (2023)
- [CSL: A Large-scale Chinese Scientific Literature Dataset](https://mlsanity.com/paper/2209.05034/) (2022)
- [FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark](https://mlsanity.com/paper/2107.07498/) (2021)
- [DTrOCR: Decoder-only Transformer for Optical Character Recognition](https://mlsanity.com/paper/2308.15996/) (2023)
- [LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation](https://mlsanity.com/paper/2108.12960/) (2021)
- [NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction](https://mlsanity.com/paper/2109.03564/) (2021)
- [TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities](https://mlsanity.com/paper/2212.06385/) (2022)
- [A Corpus for Understanding and Generating Moral Stories](https://mlsanity.com/paper/2204.09438/) (2022)
- [Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue](https://mlsanity.com/paper/2409.08330/) (2024)
- [What Can String Probability Tell Us About Grammaticality?](https://mlsanity.com/paper/2510.16227/) (2025)
- [Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model](https://mlsanity.com/paper/2112.07254/) (2021)
- [SLING: Sino Linguistic Evaluation of Large Language Models](https://mlsanity.com/paper/2210.11689/) (2022)
- [Evolving Knowledge Distillation with Large Language Models and Active Learning](https://mlsanity.com/paper/2403.06414/) (2024)
- [MetaAID: A Flexible Framework for Developing Metaverse Applications via AI Technology and Human Editing](https://mlsanity.com/paper/2204.01614/) (2022)
- [Mars: Modeling Context & State Representations with Contrastive Learning for End-to-End Task-Oriented Dialog](https://mlsanity.com/paper/2210.08917/) (2022)
- [Improving Chinese Story Generation via Awareness of Syntactic Dependencies and Semantics](https://mlsanity.com/paper/2210.10618/) (2022)
- [SikuGPT: A Generative Pre-trained Model for Intelligent Information Processing of Ancient Texts from the Perspective of Digital Humanities](https://mlsanity.com/paper/2304.07778/) (2023)
- [A Cognitive Stimulation Dialogue System with Multi-source Knowledge Fusion for Elders with Cognitive Impairment](https://mlsanity.com/paper/2305.08200/) (2023)
- [Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling](https://mlsanity.com/paper/2307.08074/) (2023)
- [CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluation](https://mlsanity.com/paper/2211.11617/) (2022)
- [Entropy-Driven Pre-Tokenization for Byte-Pair Encoding](https://mlsanity.com/paper/2506.15889/) (2025)
- [Counterfactual Recipe Generation: Exploring Compositional Generalization in a Realistic Scenario](https://mlsanity.com/paper/2210.11431/) (2022)
- [NewsDialogues: Towards Proactive News Grounded Conversation](https://mlsanity.com/paper/2308.06501/) (2023)
- [A Knowledge-enhanced Two-stage Generative Framework for Medical Dialogue Information Extraction](https://mlsanity.com/paper/2307.16200/) (2023)
- [MetaAID 2.0: An Extensible Framework for Developing Metaverse Applications via Human-controllable Pre-trained Models](https://mlsanity.com/paper/2302.13173/) (2023)
- [Methods for Acquiring and Incorporating Knowledge into Stock Price Prediction: A Survey](https://mlsanity.com/paper/2308.04947/) (2023)

## Abstract (arXiv)

Existing works, including ELMO and BERT, have revealed the importance of pre-training for NLP tasks. While there does not exist a single pre-training model that works best in all cases, it is of necessity to develop a framework that is able to deploy various pre-training models efficiently. For this purpose, we propose an assemble-on-demand pre-training toolkit, namely Universal Encoder Representations (UER). UER is loosely coupled, and encapsulated with rich modules. By assembling modules on demand, users can either reproduce a state-of-the-art pre-training model or develop a pre-training model that remains unexplored. With UER, we have built a model zoo, which contains pre-trained models based on different corpora, encoders, and targets (objectives). With proper pre-trained models, we could achieve new state-of-the-art results on a range of downstream datasets.