mlsanityData to 14 Oct 2026

Enhancing Romanian Offensive Language Detection through Knowledge Distillation, Multi-Task Learning, and Data Augmentation

arXiv 2409.20498 ↗First posted 30 Sep 2024cs.CLcited by 1

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

This paper highlights the significance of natural language processing (NLP) within artificial intelligence, underscoring its pivotal role in comprehending and modeling human language. Recent advancements in NLP, particularly in conversational bots, have garnered substantial attention and adoption among developers. This paper explores advanced methodologies for attaining smaller and more efficient NLP models. Specifically, we employ three key approaches: (1) training a Transformer-based neural network to detect offensive language, (2) employing data augmentation and knowledge distillation techniques to increase performance, and (3) incorporating multi-task learning with knowledge distillation and teacher annealing using diverse datasets to enhance efficiency. The culmination of these methods has yielded demonstrably improved outcomes.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 17 of its 38 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

17 of 38 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  5. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  6. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  7. 2021A Survey of Data Augmentation Approaches for NLPcited by 409
  8. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 207
  9. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125
  10. 2018Conditional BERT Contextual Augmentationcited by 107
  11. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  12. 2021Annealing Knowledge Distillationcited by 61
5 more
  1. 2020Knowledge Distillation for Multi-task Learningcited by 51
  2. 2020Noisy Self-Knowledge Distillation for Text Summarizationcited by 31
  3. 2021AngryBERT: Joint Learning Target and Emotion for Hate Speech Detectioncited by 19
  4. 2021Distilling the Knowledge of Romanian BERTs Using Multiple Teacherscited by 13
  5. 2021Clustering Word Embeddings with Self-Organizing Maps. Application on LaRoSeDa -- A Large Romanian Sentiment Data Setcited by 8

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024RoLargeSum: A Large Dialect-Aware Romanian News Dataset for Summary, Headline, and Keyword Generationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2021Neural Models for Offensive Language Detection
  2. 2022Rethinking Offensive Text Detection as a Multi-Hop Reasoning Problem
  3. 2021The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
  4. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalities
  5. 2026Natural Language Processing: A Comprehensive Practical Guide from Tokenisation to RLHF
  6. 2021Understanding Transformers for Bot Detection in Twitter
  7. 2024Towards Generalized Offensive Language Identification
  8. 2024A Survey on Symbolic Knowledge Distillation of Large Language Models