mlsanityData to 14 Oct 2026

Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Results

arXiv 2303.04715 ↗First posted 8 Mar 2023cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

In this paper we present the multilingual language model BLOOM-zh that features enhanced support for Traditional Chinese. BLOOM-zh has its origins in the open-source BLOOM models presented by BigScience in 2022. Starting from released models, we extended the pre-training of BLOOM by additional 7.4 billion tokens in Traditional Chinese and English covering a variety of domains such as news articles, books, encyclopedias, educational materials as well as spoken language. In order to show the properties of BLOOM-zh, both existing and newly created benchmark scenarios are used for evaluating the performance. BLOOM-zh outperforms its predecessor on most Traditional Chinese benchmarks while maintaining its English capability. We release all our models to the research community.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 8 Mar 2023.

Builds on

49 of 64 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
37 more
  1. 2016Pointer Sentinel Mixture Modelscited by 3,096
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2018Datasheets for Datasetscited by 1,386
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2018Model Cards for Model Reportingcited by 1,080
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  13. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  14. 2021Ethical and social risks of harm from Language Modelscited by 966
  15. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  16. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  17. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  18. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  19. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  20. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  21. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  22. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  23. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  24. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  25. 2018Gender Bias in Coreference Resolutioncited by 442
  26. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  27. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  28. 2021Challenges in Detoxifying Language Modelscited by 169
  29. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  30. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  31. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  32. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  33. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  34. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  35. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 76
  36. 2018DRCD: a Chinese Machine Reading Comprehension Datasetcited by 57
  37. 2022Adapting BigScience Multilingual Model to Unseen Languagescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 3; 2026: 1.

10 papers.

  1. 2023Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suitecited by 14
  2. 2024An Improved Traditional Chinese Evaluation Suite for Foundation Modelcited by 13
  3. 2025Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinesecited by 7
  4. 2024Legal Documents Drafting with Fine-Tuned Pre-Trained Large Language Modelcited by 5
  5. 2024Breeze-7B Technical Reportcited by 4
  6. 2025UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settingscited by 1
  7. 2025Measuring Hong Kong Massive Multi-Task Language Understandingcited by 1
  8. 2024Hyacinth6B: A large language model for Traditional Chinesecited by 0
  9. 2024Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embeddingcited by 0
  10. 2026Taiwan Safety Benchmark and Breeze Guard: Toward Trustworthy AI for Taiwanese Mandarincited by 0