mlsanityData to 14 Oct 2026

TCBERT: A Technical Report for Chinese Topic Classification BERT

arXiv 2211.11304 ↗First posted 21 Nov 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Bidirectional Encoder Representations from Transformers or BERT~citedevlin-etal-2019-bert has been one of the base models for various NLP tasks due to its remarkable performance. Variants customized for different languages and tasks are proposed to further improve the performance. In this work, we investigate supervised continued pre-training~citegururangan-etal-2020-dont on BERT for Chinese topic classification task. Specifically, we incorporate prompt-based learning and contrastive learning into the pre-training. To adapt to the task of Chinese topic classification, we collect around 2.1M Chinese data spanning various topics. The pre-trained Chinese Topic Classification BERTs (TCBERTs) with different parameter sizes are open-sourced at urlhttps://huggingface.co/IDEA-CCNL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

44 of 51 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
32 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  2. 2019Language Models as Knowledge Bases?cited by 1,936
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  7. 2019How Can We Know What Language Models Know?cited by 959
  8. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  9. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  10. 2021GPT Understands, Toocited by 716
  11. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  12. 2019CamemBERT: a Tasty French Language Modelcited by 368
  13. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  14. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 266
  15. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  16. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  17. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  18. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  19. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  20. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  21. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  22. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  23. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 166
  24. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  25. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  26. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  27. 2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
  28. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  29. 2021CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understandingcited by 50
  30. 2021Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuningcited by 46
  31. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  32. 2020Multi-task learning for natural language processing in the 2020s: where are we going?cited by 27

Built on by

Not cited yet.