# DynaSent: A Dynamic Benchmark for Sentiment Analysis

arXiv: https://arxiv.org/abs/2012.15349 · HTML: https://mlsanity.com/paper/2012.15349/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2020.

## Built on by (most cited first, 30 of 63)

- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [Generative Language Models Exhibit Social Identity Biases](https://mlsanity.com/paper/2310.15819/) (2023)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation](https://mlsanity.com/paper/2104.08678/) (2021)
- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://mlsanity.com/paper/2201.08277/) (2022)
- [Interactive Model Cards: A Human-Centered Approach to Model Documentation](https://mlsanity.com/paper/2205.02894/) (2022)
- [How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks](https://mlsanity.com/paper/2303.00293/) (2023)
- [Human-Adversarial Visual Question Answering](https://mlsanity.com/paper/2106.02280/) (2021)
- [A Close Look into the Calibration of Pre-trained Language Models](https://mlsanity.com/paper/2211.00151/) (2022)
- [Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking](https://mlsanity.com/paper/2106.06052/) (2021)
- [Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information](https://mlsanity.com/paper/2301.05036/) (2023)
- [Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hate](https://mlsanity.com/paper/2108.05921/) (2021)
- [AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge](https://mlsanity.com/paper/2412.13670/) (2024)
- [On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study](https://mlsanity.com/paper/2106.00872/) (2021)
- [Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants](https://mlsanity.com/paper/2112.09062/) (2021)
- [Analyzing Dynamic Adversarial Training Data in the Limit](https://mlsanity.com/paper/2110.08514/) (2021)
- [Under the Surface: Tracking the Artifactuality of LLM-Generated Data](https://mlsanity.com/paper/2401.14698/) (2024)
- [What's the Meaning of Superhuman Performance in Today's NLU?](https://mlsanity.com/paper/2305.08414/) (2023)
- [Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective](https://mlsanity.com/paper/2405.16747/) (2024)
- [Improving Model Evaluation using SMART Filtering of Benchmark Datasets](https://mlsanity.com/paper/2410.20245/) (2024)
- [Effective Human-AI Teams via Learned Natural Language Rules and Onboarding](https://mlsanity.com/paper/2311.01007/) (2023)
- [Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation](https://mlsanity.com/paper/2403.12075/) (2024)
- [Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)](https://mlsanity.com/paper/2505.21091/) (2025)
- [Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection](https://mlsanity.com/paper/2304.13148/) (2023)

## Abstract (arXiv)

We introduce DynaSent ('Dynamic Sentiment'), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis. DynaSent combines naturally occurring sentences with sentences created using the open-source Dynabench Platform, which facilities human-and-model-in-the-loop dataset creation. DynaSent has a total of 121,634 sentences, each validated by five crowdworkers, and its development and test splits are designed to produce chance performance for even the best models we have been able to develop; when future models solve this task, we will use them to create DynaSent version 2, continuing the dynamic evolution of this benchmark. Here, we report on the dataset creation effort, focusing on the steps we took to increase quality and reduce artifacts. We also present evidence that DynaSent's Neutral category is more coherent than the comparable category in other benchmarks, and we motivate training models from scratch for each round over successive fine-tuning.