# READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises

arXiv: https://arxiv.org/abs/2302.07324 · HTML: https://mlsanity.com/paper/2302.07324/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 14 Feb 2023.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Combating Adversarial Misspellings with Robust Word Recognition](https://mlsanity.com/paper/1905.11268/) (2019)
- [On Adversarial Examples for Character-Level Neural Machine Translation](https://mlsanity.com/paper/1806.09030/) (2018)
- [A Span-Extraction Dataset for Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1810.07366/) (2018)
- [OpenAttack: An Open-source Textual Adversarial Attack Toolkit](https://mlsanity.com/paper/2009.09191/) (2020)
- [It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations](https://mlsanity.com/paper/2005.04364/) (2020)
- [CharBERT: Character-aware Pre-trained Language Model](https://mlsanity.com/paper/2011.01513/) (2020)
- [Robust Encodings: A Framework for Combating Adversarial Typos](https://mlsanity.com/paper/2005.01229/) (2020)
- [SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check](https://mlsanity.com/paper/2004.14166/) (2020)
- [Investigating African-American Vernacular English in Transformer-Based Text Generation](https://mlsanity.com/paper/2010.02510/) (2020)
- [VALUE: Understanding Dialect Disparity in NLU](https://mlsanity.com/paper/2204.03031/) (2022)
- [Meta-Learning for Domain Generalization in Semantic Parsing](https://mlsanity.com/paper/2010.11988/) (2020)
- [A Pilot Study for Chinese SQL Semantic Parsing](https://mlsanity.com/paper/1909.13293/) (2019)
- [NoiseQA: Challenge Set Evaluation for User-Centric Question Answering](https://mlsanity.com/paper/2102.08345/) (2021)
- [GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors](https://mlsanity.com/paper/1911.12893/) (2019)
- [Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding](https://mlsanity.com/paper/2004.14870/) (2020)
- [Misspelling Oblivious Word Embeddings](https://mlsanity.com/paper/1905.09755/) (2019)
- [BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance](https://mlsanity.com/paper/1910.07181/) (2019)
- [Benchmarking Robustness of Machine Reading Comprehension Models](https://mlsanity.com/paper/2004.14004/) (2020)

## Built on by (most cited first, 1 of 1)

- [Evaluating the Robustness of Japanese LLMs to IME-Related and Typographical Errors](https://mlsanity.com/paper/2610.01241/) (2026)

## Abstract (arXiv)

For many real-world applications, the user-generated inputs usually contain various noises due to speech recognition errors caused by linguistic variations1 or typographical errors (typos). Thus, it is crucial to test model performance on data with realistic input noises to ensure robustness and fairness. However, little study has been done to construct such benchmarks for Chinese, where various language-specific input noises happen in the real world. In order to fill this important gap, we construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input. We designed our annotation pipeline to maximize diversity, for example by instructing the annotators to use diverse input method editors (IMEs) for keyboard noises and recruiting speakers from diverse dialectical groups for speech noises. We experiment with a series of strong pretrained language models as well as robust training methods, we find that these models often suffer significant performance drops on READIN even with robustness methods like data augmentation. As the first large-scale attempt in creating a benchmark with noises geared towards user-generated inputs, we believe that READIN serves as an important complement to existing Chinese NLP benchmarks. The source code and dataset can be obtained from https://github.com/thunlp/READIN.