# ExplainaBoard: An Explainable Leaderboard for NLP

arXiv: https://arxiv.org/abs/2104.06387 · HTML: https://mlsanity.com/paper/2104.06387/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 57 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](https://mlsanity.com/paper/1909.09251/) (2019)
- [FLERT: Document-Level Features for Named Entity Recognition](https://mlsanity.com/paper/2011.06993/) (2020)
- [compare-mt: A Tool for Holistic Comparison of Language Generation Systems](https://mlsanity.com/paper/1903.07926/) (2019)
- [Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa](https://mlsanity.com/paper/2104.04986/) (2021)
- [Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study](https://mlsanity.com/paper/2001.03844/) (2020)
- [Interpretable Multi-dataset Evaluation for Named Entity Recognition](https://mlsanity.com/paper/2011.06854/) (2020)
- [Towards More Fine-grained and Reliable NLP Performance Prediction](https://mlsanity.com/paper/2102.05486/) (2021)
- [A Closer Look at Data Bias in Neural Extractive Summarization Models](https://mlsanity.com/paper/1909.13705/) (2019)

## Built on by (most cited first, 30 of 50)

- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [SpanNER: Named Entity Re-/Recognition as Span Prediction](https://mlsanity.com/paper/2106.00641/) (2021)
- [Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking](https://mlsanity.com/paper/2106.06052/) (2021)
- [Calibrating LLM-Based Evaluator](https://mlsanity.com/paper/2309.13308/) (2023)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Multi-Dimensional Evaluation of Text Summarization with In-Context Learning](https://mlsanity.com/paper/2306.01200/) (2023)
- [Towards LLM-guided Causal Explainability for Black-box Text Classifiers](https://mlsanity.com/paper/2309.13340/) (2023)
- [MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers](https://mlsanity.com/paper/2109.00799/) (2021)
- [MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://mlsanity.com/paper/2210.12391/) (2022)
- [Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements](https://mlsanity.com/paper/2210.01970/) (2022)
- [DABERT: Dual Attention Enhanced BERT for Semantic Matching](https://mlsanity.com/paper/2210.03454/) (2022)
- [Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflicts](https://mlsanity.com/paper/2302.12530/) (2023)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)
- [Predicting Fine-Tuning Performance with Probing](https://mlsanity.com/paper/2210.07352/) (2022)
- [BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation](https://mlsanity.com/paper/2109.06850/) (2021)
- [SketchXAI: A First Look at Explainability for Human Sketches](https://mlsanity.com/paper/2304.11744/) (2023)
- [Numerical reasoning in machine reading comprehension tasks: are we there yet?](https://mlsanity.com/paper/2109.08207/) (2021)
- [GlobalBench: A Benchmark for Global Progress in Natural Language Processing](https://mlsanity.com/paper/2305.14716/) (2023)
- [Comateformer: Combined Attention Transformer for Semantic Sentence Matching](https://mlsanity.com/paper/2412.07220/) (2024)
- [DataLab: A Platform for Data Analysis and Intervention](https://mlsanity.com/paper/2202.12875/) (2022)
- [IFAN: An Explainability-Focused Interaction Framework for Humans and NLP Models](https://mlsanity.com/paper/2303.03124/) (2023)

## Abstract (arXiv)

With the rapid development of NLP research, leaderboards have emerged as one tool to track the performance of various systems on various NLP tasks. They are effective in this goal to some extent, but generally present a rather simplistic one-dimensional view of the submitted systems, communicated only through holistic accuracy numbers. In this paper, we present a new conceptualization and implementation of NLP evaluation: the ExplainaBoard, which in addition to inheriting the functionality of the standard leaderboard, also allows researchers to (i) diagnose strengths and weaknesses of a single system (e.g.~what is the best-performing system bad at?) (ii) interpret relationships between multiple systems. (e.g.~where does system A outperform system B? What if we combine systems A, B, and C?) and (iii) examine prediction results closely (e.g.~what are common errors made by multiple systems, or in what contexts do particular errors occur?). So far, ExplainaBoard covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks. ExplainaBoard keeps updated and is recently upgraded by supporting (1) multilingual multi-task benchmark, (2) meta-evaluation, and (3) more complicated task: machine translation, which reviewers also suggested. We not only released an online platform on the website urlhttp://explainaboard.nlpedia.ai/ but also make our evaluation tool an API with MIT Licence at Github urlhttps://github.com/neulab/explainaBoard and PyPi urlhttps://pypi.org/project/interpret-eval/ that allows users to conveniently assess their models offline. We additionally release all output files from systems that we have run or collected to motivate "output-driven" research in the future.