mlsanityData to 13 Oct 2026

KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding

arXiv 2101.11363 ↗First posted 27 Jan 2021cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

A Lite BERT (ALBERT) has been introduced to scale up deep bidirectional representation learning for natural languages. Due to the lack of pretrained ALBERT models for Korean language, the best available practice is the multilingual model or resorting back to the any other BERT-based model. In this paper, we develop and pretrain KoreALBERT, a monolingual ALBERT model specifically for Korean language understanding. We introduce a new training objective, namely Word Order Prediction (WOP), and use alongside the existing MLM and SOP criteria to the same architecture and model parameters. Despite having significantly fewer model parameters (thus, quicker to train), our pretrained KoreALBERT outperforms its BERT counterpart on 6 different NLU tasks. Consistent with the empirical results in English by Lan et al., KoreALBERT seems to improve downstream task performance involving multi-sentence encoding for Korean language. The pretrained KoreALBERT is publicly available to encourage research and application development for Korean NLP.

Key facts. Cited by 9 papers in the map (data through 13 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 27 Jan 2021.

Builds on

21 of 30 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,981
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,979
9 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,546
  2. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 854
  3. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 833
  4. 2019CamemBERT: a Tasty French Language Modelcited by 368
  5. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  6. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  7. 2019Multilingual is not enough: BERT for Finnishcited by 118
  8. 2019BERTje: A Dutch BERT Modelcited by 109
  9. 2020KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understandingcited by 37

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 2; 2024: 1; 2025: 1.

9 papers.

  1. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  2. 2023Sabi'a: Portuguese Large Language Modelscited by 19
  3. 2022Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-trainingcited by 6
  4. 2021A Survey on Awesome Korean NLP Datasetscited by 3
  5. 2022MonoByte: A Pool of Monolingual Byte-level Language Modelscited by 1
  6. 2025Sovereign Large Language Models: Advantages, Strategy and Regulationscited by 1
  7. 2022Factorization Approach for Sparse Spatio-Temporal Brain-Computer Interfacecited by 0
  8. 2023Exploring acceptance of autonomous vehicle policies using KeyBERT and SNA: Targeting engineering studentscited by 0
  9. 2024KPC-cF: Aspect-Based Sentiment Analysis via Implicit-Feature Alignment with Corpus Filteringcited by 0