mlsanityData to 7 Oct 2026

Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Dataset

arXiv 2507.03378 ↗First posted 4 Jul 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Although LLMs have made significant progress in various languages, there are still concerns about their effectiveness with low-resource agglutinative languages compared to languages such as English. In this study, we focused on Korean, a language known for its complex sentence endings, and evaluated LLMs on this challenging aspect. We introduce the Korean Sentence Endings (KoSEnd) dataset, which includes 3,000 sentences, each annotated for the naturalness of 15 sentence ending forms. These were collected from diverse sources to cover a range of contexts. We evaluated 11 LLMs to assess their understanding of Korean sentence endings, analyzing them based on parameter count and prediction consistency. Notably, we found that informing models about the possibility of missing sentence endings improved performance, highlighting the impact of explicitly considering certain linguistic features.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 4 Jul 2025.

Builds on

33 of 59 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  12. 2022Is GPT-3 a Good Data Annotator?cited by 208
21 more
  1. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  2. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  3. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  4. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  5. 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
  6. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  7. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  8. 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 65
  9. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 61
  10. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  11. 2019KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehensioncited by 43
  12. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  13. 2020KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understandingcited by 37
  14. 2022KOBEST: Korean Balanced Evaluation of Significant Taskscited by 37
  15. 2024KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledgecited by 33
  16. 2020Linguistic Profiling of a Neural Language Modelcited by 23
  17. 2024Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmarkcited by 23
  18. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  19. 2022Towards standardizing Korean Grammatical Error Correction: Datasets and Annotationcited by 13
  20. 2024Does Incomplete Syntax Influence Korean Language Model? Focusing on Word Order and Case Markerscited by 6
  21. 2024KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmarkcited by 2

Built on by

Not cited yet.