# RoBERTa-wwm-ext Fine-Tuning for Chinese Text Classification

arXiv: https://arxiv.org/abs/2103.00492 · HTML: https://mlsanity.com/paper/2103.00492/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 7 references that are in the map.
- First posted on arXiv 24 Feb 2021.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)

## Built on by (most cited first, 2 of 2)

- [Continual Graph Convolutional Network for Text Classification](https://mlsanity.com/paper/2304.04152/) (2023)
- [A Comparative Study of LLM Prompting and Fine-Tuning for Cross-genre Authorship Attribution on Chinese Lyrics](https://mlsanity.com/paper/2511.21930/) (2025)

## Abstract (arXiv)

Bidirectional Encoder Representations from Transformers (BERT) have shown to be a promising way to dramatically improve the performance across various Natural Language Processing tasks [Devlin et al., 2019]. Meanwhile, progress made over the past few years by various Neural Net-work has also proved the effectiveness of Neural Network in the field of Natural Language Processing. In this project, RoBERTa-wwm-ext [Cui et al., 2019] pre-train language model was adopted and fine-tuned for Chinese text classification. The models were able to classify Chinese texts into two categories, containing descriptions of legal behavior and descriptions of illegal behavior. Four different models are also proposed in the paper. Those models will use RoBERTa-wwm-extas their embedding layer and feed the embedding into different neural networks. The motivation be-hind proposing these models is straightforward. By introducing complex output layer architecture, the overall performance of the models could be improved. All the models were trained on a data set derived from Chinese public court records, and the performance of different models were compared.The experiment shows that the performance of pro-posed models failed to beat the original RoBERTa-wwm-ext model in terms of accuracy and training efficiency.