mlsanityData to 7 Oct 2026

Predicting Performance for Natural Language Processing Tasks

arXiv 2005.00870 ↗First posted 2 May 2020cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Given the complexity of combinations of tasks, languages, and domains in natural language processing (NLP) research, it is computationally prohibitive to exhaustively test newly proposed models on each possible experimental setting. In this work, we attempt to explore the possibility of gaining plausible judgments of how well an NLP model can perform under an experimental setting, without actually training or testing the model. To do so, we build regression models to predict the evaluation score of an NLP experiment given the experimental settings as input. Experimenting on 9 different NLP tasks, we find that our predictors can produce meaningful predictions over unseen languages and different modeling architectures, outperforming reasonable baselines as well as human experts. Going further, we outline how our predictor can be used to find a small subset of representative experiments that should be run in order to obtain plausible predictions for all other experimental settings.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 17; 2023: 3; 2024: 9; 2025: 9; 2026: 3.

49 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  3. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performancecited by 51
  4. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  5. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  6. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  7. 2023How Predictable Are Large Language Model Capabilities? A Case Study on BIG-benchcited by 34
  8. 2021Towards More Fine-grained and Reliable NLP Performance Predictioncited by 26
  9. 2021Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Fillingcited by 25
  10. 2021DirectProbe: Studying Representations without Classifierscited by 25
  11. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  12. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
37 more
  1. 2024URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Basecited by 21
  2. 2022Multi Task Learning For Zero Shot Performance Prediction of Multilingual Modelscited by 20
  3. 2024ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Modelscited by 20
  4. 2022Predicting Fine-Tuning Performance with Probingcited by 13
  5. 2022Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systemscited by 12
  6. 2022Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfercited by 11
  7. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  8. 2022Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languagescited by 10
  9. 2025Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Mattercited by 10
  10. 2024What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinementcited by 9
  11. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  12. 2021Towards More Equitable Question Answering Systems: How Much More Data Do You Need?cited by 8
  13. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  14. 2022Evidence > Intuition: Transferability Estimation for Encoder Selectioncited by 7
  15. 2024A Reproducibility Study on Quantifying Language Similarity: The Impact of Missing Values in the URIEL Knowledge Basecited by 7
  16. 2024Influential Language Data Selection via Gradient Trajectory Pursuitcited by 7
  17. 2021Model Compression for Domain Adaptation through Causal Effect Estimationcited by 6
  18. 2022Assessing Out-of-Domain Language Model Performance from Few Examplescited by 6
  19. 2022Embarrassingly Simple Performance Prediction for Abductive Natural Language Inferencecited by 5
  20. 2022BERTops: Studying BERT Representations under a Topological Lenscited by 5
  21. 2022Rethinking Round-Trip Translation for Machine Translation Evaluationcited by 4
  22. 2022Towards Automated Error Analysis: Learning to Characterize Errorscited by 3
  23. 2022Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classificationcited by 3
  24. 2025Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+cited by 3
  25. 2022DaLC: Domain Adaptation Learning Curve Prediction for Neural Machine Translationcited by 2
  26. 2022Sort by Structure: Language Model Ranking as Dependency Probingcited by 2
  27. 2022Detecting Languages Unintelligible to Multilingual Models through Local Structure Probescited by 2
  28. 2025We Need to Measure Data Diversity in NLP -- Better and Broadercited by 2
  29. 2025Zero-Shot Performance Prediction for Probabilistic Scaling Lawscited by 2
  30. 2025Anticipatory Evaluation of Language Modelscited by 1
  31. 2025Alchemist: Unlocking Efficiency in Text-to-Image Model Training via Meta-Gradient Data Selectioncited by 1
  32. 2023X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similaritycited by 0
  33. 2024Third-Party Language Model Performance Prediction from Instructioncited by 0
  34. 2025Can We Reliably Rank Model Performance across Domains without Labeled Data?cited by 0
  35. 2026Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruningcited by 0
  36. 2026Typological Feature Prediction with Large Language Models: An In-Context Learning Approachcited by 0
  37. 2026Language Distances are Practical for Equitable Cross-Lingual Transfercited by 0