mlsanityData to 14 Oct 2026

Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean

arXiv 2506.01237 ↗First posted 2 Jun 2025cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

We introduce the underlineKorean underlineGrammar underlineEvaluation BenchunderlineMark (KoGEM), designed to assess the linguistic competence of LLMs and humans in Korean. KoGEM consists of 1.5k multiple-choice QA pairs covering five main categories and 16 subcategories. The zero-shot evaluation of 27 LLMs of various sizes and types reveals that while LLMs perform remarkably well on straightforward tasks requiring primarily definitional knowledge, they struggle with tasks that demand the integration of real-world experiential knowledge, such as phonological rules and pronunciation. Furthermore, our in-depth analysis suggests that incorporating such experiential knowledge could enhance the linguistic competence of LLMs. With KoGEM, we not only highlight the limitations of current LLMs in linguistic competence but also uncover hidden facets of LLMs in linguistic competence, paving the way for enhancing comprehensive language understanding. Our code and dataset are available at: https://github.com/SungHo3268/KoGEM.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 61 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

23 of 61 references.

References · 23

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  9. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  10. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  11. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  12. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
11 more
  1. 2022The Geometry of Multilingual Language Model Representationscited by 95
  2. 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 66
  3. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 53
  4. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  5. 2022Prompting Language Models for Linguistic Structurecited by 39
  6. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  7. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilitiescited by 18
  8. 2024Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Koreancited by 14
  9. 2022Towards standardizing Korean Grammatical Error Correction: Datasets and Annotationcited by 13
  10. 2019Extending Machine Language Models toward Human-Level Language Understandingcited by 7
  11. 2024Large Language Models for Psycholinguistic Plausibility Pretestingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Modelscited by 0
  2. 2026Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinesecited by 0