mlsanityData to 7 Oct 2026

What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

arXiv 2109.04650 ↗First posted 10 Sep 2021cs.CLcited by 75

Not on the map. Why?

Abstract (arXiv)

GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

37 of 56 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
25 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  4. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  5. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  6. 2020Recipes for building an open-domain chatbotcited by 593
  7. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  8. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  9. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  10. 2019CamemBERT: a Tasty French Language Modelcited by 368
  11. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  12. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  13. 2021Generating Datasets with Pretrained Language Modelscited by 192
  14. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  15. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 164
  16. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 159
  17. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  18. 2020PhoBERT: Pre-trained language models for Vietnamesecited by 148
  19. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  20. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  21. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  22. 2019KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehensioncited by 43
  23. 2020Controlling Style in Generated Dialoguecited by 38
  24. 2020An Empirical Study of Tokenization Strategies for Various Korean NLP Taskscited by 33
  25. 2019Multi-Modal Generative Adversarial Network for Short Product Title Generation in Mobile E-Commercecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 19; 2023: 25; 2024: 16; 2025: 4; 2026: 3.

75 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  4. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  7. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  8. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  9. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  10. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 102
  11. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  12. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
38 more
  1. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  2. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  3. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  4. 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
  5. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  6. 2022Keep Me Updated! Memory Management in Long-term Conversationscited by 62
  7. 2023A Bibliometric Review of Large Language Models Research from 2017 to 2023cited by 56
  8. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  9. 2023Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Datacited by 48
  10. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  11. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  12. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  13. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  14. 2022Prompt Injection: Parameterization of Fixed Inputscited by 35
  15. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  16. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  17. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  18. 2023ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Eventscited by 31
  19. 2024History, Development, and Principles of Large Language Models-An Introductory Surveycited by 30
  20. 2023ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Predictioncited by 27
  21. 2021Scaling Law for Recommendation Models: Towards General-purpose User Representationscited by 26
  22. 2022Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Modelscited by 26
  23. 2023KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applicationcited by 24
  24. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Modelscited by 22
  25. 2023Sabi'a: Portuguese Large Language Modelscited by 19
  26. 2024HyperCLOVA X Technical Reportcited by 18
  27. 2024Understanding the Impact of Long-Term Memory on Self-Disclosure with Large Language Model-Driven Chatbots for Public Health Interventioncited by 15
  28. 2024Navigating the Path of Writing: Outline-guided Text Generation with Large Language Modelscited by 15
  29. 2023Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defensecited by 13
  30. 2023SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaborationcited by 13
  31. 2024LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Surveycited by 12
  32. 2023Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Resultscited by 10
  33. 2024Fairness Definitions in Language Models Explainedcited by 10
  34. 2021PAGnol: An Extra-Large French Generative Modelcited by 9
  35. 2021Contrastive Fine-grained Class Clustering via Generative Adversarial Networkscited by 9
  36. 2022Building a Personalized Dialogue System with Prompt-Tuningcited by 9
  37. 2023GLM-Dialog: Noise-tolerant Pre-training for Knowledge-grounded Dialogue Generationcited by 9
  38. 2024Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina PT* Familycited by 8

50 most cited of 75 All 75 (JSON)