# Hatevolution: What Static Benchmarks Don't Tell Us

arXiv: https://arxiv.org/abs/2506.12148 · HTML: https://mlsanity.com/paper/2506.12148/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Builds on (most cited first, 30 of 31)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [Time Waits for No One! Analysis and Challenges of Temporal Misalignment](https://mlsanity.com/paper/2111.07408/) (2021)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists](https://mlsanity.com/paper/2203.09192/) (2022)
- [Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media](https://mlsanity.com/paper/2104.08116/) (2021)
- [On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research](https://mlsanity.com/paper/2304.12397/) (2023)
- [Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift](https://mlsanity.com/paper/1912.01072/) (2019)
- [NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms](https://mlsanity.com/paper/2402.12261/) (2024)
- [Lifelong Learning of Hate Speech Classification on Social Media](https://mlsanity.com/paper/2106.02821/) (2021)
- [HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysis](https://mlsanity.com/paper/2202.03612/) (2022)
- [SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research](https://mlsanity.com/paper/2310.14757/) (2023)
- [Building for Tomorrow: Assessing the Temporal Persistence of Text Classifiers](https://mlsanity.com/paper/2205.05435/) (2022)

## Abstract (arXiv)

Language changes over time, including in the hate speech domain, which evolves quickly following social dynamics and cultural shifts. While NLP research has investigated the impact of language evolution on model training and has proposed several solutions for it, its impact on model benchmarking remains under-explored. Yet, hate speech benchmarks play a crucial role to ensure model safety. In this paper, we empirically evaluate the robustness of 20 language models across two evolving hate speech experiments, and we show the temporal misalignment between static and time-sensitive evaluations. Our findings call for time-sensitive linguistic benchmarks in order to correctly and reliably evaluate language models in the hate speech domain.