Transformer-based Korean Pretrained Language Models: A Survey on Three Years of Progress
Not on the map. Why?
Abstract (arXiv)
With the advent of Transformer, which was used in translation models in 2017, attention-based architectures began to attract attention. Furthermore, after the emergence of BERT, which strengthened the NLU-specific encoder part, which is a part of the Transformer, and the GPT architecture, which strengthened the NLG-specific decoder part, various methodologies, data, and models for learning the Pretrained Language Model began to appear. Furthermore, in the past three years, various Pretrained Language Models specialized for Korean have appeared. In this paper, we intend to numerically and qualitatively compare and analyze various Korean PLMs released to the public.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 25 Nov 2021.
Builds on
23 of 32 references.
References · 23
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
11 more
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
- 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
- 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
- 2018Scaling Neural Machine Translationcited by 204
- 2019The Evolved Transformercited by 180
- 2021KLUE: Korean Language Understanding Evaluationcited by 113
- 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
- 2020BEEP! Korean Corpus of Online News Comments for Toxic Speech Detectioncited by 39
- 2020KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understandingcited by 37
Built on by
Citing papers per year: 2024: 1.
1 paper.