# What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

arXiv: https://arxiv.org/abs/2005.14299 · HTML: https://mlsanity.com/paper/2005.14299/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2020.

## Built on by (most cited first, 1 of 1)

- [Architectures of Meaning, A Systematic Corpus Analysis of NLP Systems](https://mlsanity.com/paper/2107.08124/) (2021)

## Abstract (arXiv)

SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.