mlsanityData to 7 Oct 2026

Towards More Fine-grained and Reliable NLP Performance Prediction

arXiv 2102.05486 ↗First posted 10 Feb 2021cs.CL cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Performance prediction, the task of estimating a system's performance without performing experiments, allows us to reduce the experimental burden caused by the combinatorial explosion of different datasets, languages, tasks, and models. In this paper, we make two contributions to improving performance prediction for NLP tasks. First, we examine performance predictors not only for holistic measures of accuracy like F1 or BLEU but also fine-grained performance measures such as accuracy over individual classes of examples. Second, we propose methods to understand the reliability of a performance prediction model from two angles: confidence intervals and calibration. We perform an analysis of four types of NLP tasks, and both demonstrate the feasibility of fine-grained performance prediction and the necessity to perform reliability analysis for performance prediction methods in the future. We make our code publicly available: urlhttps://github.com/neulab/Reliable-NLPPP

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 22 of its 51 references that are in the map. First posted on arXiv 10 Feb 2021.

Builds on

22 of 51 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  8. 2016Neural Architectures for Named Entity Recognitioncited by 548
  9. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  10. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  11. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  12. 2014A Convolutional Neural Network for Modelling Sentencescited by 163
10 more
  1. 2015Visualizing and Understanding Recurrent Networkscited by 158
  2. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  3. 2019Generalized Data Augmentation for Low-Resource Translationcited by 53
  4. 2020Predicting Performance for Natural Language Processing Taskscited by 49
  5. 2019compare-mt: A Tool for Holistic Comparison of Language Generation Systemscited by 43
  6. 2020Rethinking Generalization of Neural Models: A Named Entity Recognition Case Studycited by 35
  7. 2020Interpretable Multi-dataset Evaluation for Named Entity Recognitioncited by 27
  8. 2019GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognitioncited by 18
  9. 2017Modeling Source Syntax for Neural Machine Translationcited by 12
  10. 2020RethinkCWS: Is Chinese Word Segmentation a Solved Task?cited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 11; 2023: 4; 2024: 3; 2025: 3; 2026: 1.

26 papers.

Sort:

By area:

  1. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  2. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  3. 2023How Predictable Are Large Language Model Capabilities? A Case Study on BIG-benchcited by 34
  4. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  5. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  6. 2022Multi Task Learning For Zero Shot Performance Prediction of Multilingual Modelscited by 20
  7. 2024ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Modelscited by 20
  8. 2022Predicting Fine-Tuning Performance with Probingcited by 13
  9. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  10. 2022Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languagescited by 10
  11. 2023How to Estimate Model Transferability of Pre-Trained Speech Models?cited by 10
  12. 2022Disentangling Uncertainty in Machine Translation Evaluationcited by 9
14 more
  1. 2022Evidence > Intuition: Transferability Estimation for Encoder Selectioncited by 7
  2. 2022Assessing Out-of-Domain Language Model Performance from Few Examplescited by 6
  3. 2022Embarrassingly Simple Performance Prediction for Abductive Natural Language Inferencecited by 5
  4. 2023Linking Surface Facts to Large-Scale Knowledge Graphscited by 5
  5. 2021Larger-Context Tagging: When and Why Does It Work?cited by 4
  6. 2022Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classificationcited by 3
  7. 2022DaLC: Domain Adaptation Learning Curve Prediction for Neural Machine Translationcited by 2
  8. 2022Sort by Structure: Language Model Ranking as Dependency Probingcited by 2
  9. 2022Detecting Languages Unintelligible to Multilingual Models through Local Structure Probescited by 2
  10. 2025Zero-Shot Performance Prediction for Probabilistic Scaling Lawscited by 2
  11. 2023Characterizing and Measuring Linguistic Dataset Driftcited by 1
  12. 2025Anticipatory Evaluation of Language Modelscited by 1
  13. 2024Third-Party Language Model Performance Prediction from Instructioncited by 0
  14. 2026Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruningcited by 0