mlsanityData to 14 Oct 2026

From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse

arXiv 2509.14712 ↗First posted 18 Sep 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Although offensive language continually evolves over time, even recent studies using LLMs have predominantly relied on outdated datasets and rarely evaluated the generalization ability on unseen texts. In this study, we constructed a large-scale dataset of contemporary political discourse and employed three refined judgments in the absence of ground truth. Each judgment reflects a representative offensive language detection method and is carefully designed for optimal conditions. We identified distinct patterns for each judgment and demonstrated tendencies of label agreement using a leave-one-out strategy. By establishing pseudo-labels as ground trust for quantitative performance assessment, we observed that a strategically designed single prompting achieves comparable performance to more resource-intensive methods. This suggests a feasible approach applicable in real-world settings with inherent constraints.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 99 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

45 of 99 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  5. 2022A Survey on In-context Learningcited by 721
  6. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  7. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 407
  8. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  9. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 271
  10. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  11. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 207
  12. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
33 more
  1. 2023Text Classification via Large Language Modelscited by 106
  2. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  3. 2022COLD: A Benchmark for Chinese Offensive Language Detectioncited by 76
  4. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  5. 2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
  6. 2020SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identificationcited by 53
  7. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 53
  8. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  9. 2023COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statementscited by 34
  10. 2024ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbationscited by 31
  11. 2024Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targetscited by 31
  12. 2024Debate-to-Write: A Persona-Driven Multi-Agent Framework for Diverse Argument Generationcited by 29
  13. 2024Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Biascited by 28
  14. 2022Chain of Explanation: New Prompting Method to Generate Higher Quality Natural Language Explanation for Implicit Hate Speechcited by 25
  15. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 23
  16. 2020Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learningcited by 21
  17. 2023Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Modelcited by 21
  18. 2024Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personascited by 16
  19. 2023JointMatch: A Unified Approach for Diverse and Collaborative Pseudo-Labeling to Semi-Supervised Text Classificationcited by 15
  20. 2022K-MHaS: A Multi-label Hate Speech Detection Dataset in Korean Online News Commentcited by 13
  21. 2023Towards a Holistic Approach: Understanding Sociodemographic Biases in NLP Models using an Interdisciplinary Lenscited by 9
  22. 2024Chain of Evidences and Evidence to Generate: Prompting for Context Grounded and Retrieval Augmented Reasoningcited by 7
  23. 2024"Define Your Terms" : Enhancing Efficient Offensive Speech Classification with Definitioncited by 7
  24. 2025The Political Ideology of Large Language Models: Measurement, Inconsistency, and Persuasive Influencecited by 7
  25. 2022Korean Online Hate Speech Dataset for Multilabel Classification: How Can Social Science Improve Dataset on Hate Speech?cited by 6
  26. 2024Untangling Hate Speech Definitions: A Semantic Componential Analysis Across Cultures and Domainscited by 6
  27. 2025Active Few-Shot Learning for Text Classificationcited by 6
  28. 2023K-HATERS: A Hate Speech Detection Corpus in Korean with Target-Specific Ratingscited by 5
  29. 2025Irony Detection, Reasoning and Understanding in Zero-shot Learningcited by 5
  30. 2023"We Demand Justice!": Towards Social Context Grounding of Political Textscited by 4
  31. 2024AustroTox: A Dataset for Target-Based Austrian German Offensive Language Detectioncited by 4
  32. 2024Don't be a Fool: Pooling Strategies in Offensive Language Detection from User-Intended Adversarial Attackscited by 2
  33. 2025Evaluating the Effectiveness of Large Language Models in Automated News Article Summarizationcited by 2

Built on by

Not cited yet.