mlsanityData to 14 Oct 2026

Exploring the Usage of Chinese Pinyin in Pretraining

arXiv 2310.04960 ↗First posted 8 Oct 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Unlike alphabetic languages, Chinese spelling and pronunciation are different. Both characters and pinyin take an important role in Chinese language understanding. In Chinese NLP tasks, we almost adopt characters or words as model input, and few works study how to use pinyin. However, pinyin is essential in many scenarios, such as error correction and fault tolerance for ASR-introduced errors. Most of these errors are caused by the same or similar pronunciation words, and we refer to this type of error as SSP(the same or similar pronunciation) errors for short. In this work, we explore various ways of using pinyin in pretraining models and propose a new pretraining method called PmBERT. Our method uses characters and pinyin in parallel for pretraining. Through delicate pretraining tasks, the characters and pinyin representation are fused, which can enhance the error tolerance for SSP errors. We do comprehensive experiments and ablation tests to explore what makes a robust phonetic enhanced Chinese language model. The experimental results on both the constructed noise-added dataset and the public error-correction dataset demonstrate that our model is more robust compared to SOTA models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

18 of 29 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  7. 2019Text Summarization with Pretrained Encoderscited by 556
  8. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  9. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  10. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
  11. 2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
  12. 2021Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checkingcited by 52
6 more
  1. 2021Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correctioncited by 19
  2. 2020Data Augmentation for Training Dialog Models Robust to Speech Recognition Errorscited by 17
  3. 2021An Approach to Improve Robustness of NLP Systems against ASR Errorscited by 8
  4. 2017The First Evaluation of Chinese Human-Computer Dialogue Technologycited by 7
  5. 2018Improving the Robustness of Speech Translationcited by 7
  6. 2022PERT: A New Solution to Pinyin to Character Conversion Taskcited by 2

Built on by

Not cited yet.