# SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research

arXiv: https://arxiv.org/abs/2310.14757 · HTML: https://mlsanity.com/paper/2310.14757/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 89 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 30 of 44)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://mlsanity.com/paper/2212.12017/) (2022)
- [SemEval-2017 Task 4: Sentiment Analysis in Twitter](https://mlsanity.com/paper/1912.00741/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Multilingual and Multi-Aspect Hate Speech Analysis](https://mlsanity.com/paper/1908.11049/) (2019)
- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021)
- [Time Waits for No One! Analysis and Challenges of Temporal Misalignment](https://mlsanity.com/paper/2111.07408/) (2021)
- [TWEETQA: A Social Media Focused Question Answering Dataset](https://mlsanity.com/paper/1907.06292/) (2019)

## Built on by (most cited first, 9 of 9)

- [AI in Mental Health: Emotional and Sentiment Analysis of Large Language Models' Responses to Depression, Anxiety, and Stress Queries](https://mlsanity.com/paper/2508.11285/) (2025)
- [A Survey of Machine Learning Models and Datasets for the Multi-label Classification of Textual Hate Speech in English](https://mlsanity.com/paper/2504.08609/) (2025)
- [WiDe-analysis: Enabling One-click Content Moderation Analysis on Wikipedia's Articles for Deletion](https://mlsanity.com/paper/2408.05655/) (2024)
- [Causal Effects of Trigger Words in Social Media Discussions: A Large-Scale Case Study about UK Politics on Reddit](https://mlsanity.com/paper/2405.10213/) (2024)
- [A Systematic Analysis on the Temporal Generalization of Language Models in Social Media](https://mlsanity.com/paper/2405.13017/) (2024)
- [Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Models](https://mlsanity.com/paper/2410.19195/) (2024)
- [Hatevolution: What Static Benchmarks Don't Tell Us](https://mlsanity.com/paper/2506.12148/) (2025)
- [Predicting Social Media User Actions: A Hybrid Approach for Common and Rare Behavior Prediction on Bluesky](https://mlsanity.com/paper/2511.17241/) (2025)
- [Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media](https://mlsanity.com/paper/2605.21338/) (2026)

## Abstract (arXiv)

Despite its relevance, the maturity of NLP for social media pales in comparison with general-purpose models, metrics and benchmarks. This fragmented landscape makes it hard for the community to know, for instance, given a task, which is the best performing model and how it compares with others. To alleviate this issue, we introduce a unified benchmark for NLP evaluation in social media, SuperTweetEval, which includes a heterogeneous set of tasks and datasets combined, adapted and constructed from scratch. We benchmarked the performance of a wide range of models on SuperTweetEval and our results suggest that, despite the recent advances in language modelling, social media remains challenging.