mlsanityData to 14 Oct 2026

Hatevolution: What Static Benchmarks Don't Tell Us

arXiv 2506.12148 ↗First posted 13 Jun 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Language changes over time, including in the hate speech domain, which evolves quickly following social dynamics and cultural shifts. While NLP research has investigated the impact of language evolution on model training and has proposed several solutions for it, its impact on model benchmarking remains under-explored. Yet, hate speech benchmarks play a crucial role to ensure model safety. In this paper, we empirically evaluate the robustness of 20 language models across two evolving hate speech experiments, and we show the temporal misalignment between static and time-sensitive evaluations. Our findings call for time-sensitive linguistic benchmarks in order to correctly and reliably evaluate language models in the hate speech domain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

31 of 52 references.

References · 31

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  6. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  7. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  8. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 407
  9. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  10. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  11. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  12. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
19 more
  1. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  2. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  3. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 217
  4. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
  5. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  6. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  7. 2022TimeLMs: Diachronic Language Models from Twittercited by 131
  8. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  9. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 88
  10. 2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 55
  11. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  12. 2023On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Researchcited by 44
  13. 2019Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shiftcited by 37
  14. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 20
  15. 2021Lifelong Learning of Hate Speech Classification on Social Mediacited by 14
  16. 2022HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysiscited by 11
  17. 2023SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Researchcited by 9
  18. 2022Building for Tomorrow: Assessing the Temporal Persistence of Text Classifierscited by 8
  19. 2023Examining Temporal Bias in Abusive Language Detectioncited by 4

Built on by

Not cited yet.