mlsanityData to 14 Oct 2026

Experimental Standards for Deep Learning in Natural Language Processing Research

arXiv 2204.06251 ↗First posted 13 Apr 2022cs.LG cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, compared to more established disciplines, a lack of common experimental standards remains an open challenge to the field at large. Starting from fundamental scientific principles, we distill ongoing discussions on experimental standards in NLP into a single, widely-applicable methodology. Following these best practices is crucial to strengthen experimental evidence, improve reproducibility and support scientific progress. These standards are further collected in a public repository to help them transparently adapt to future needs.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 71 of its 177 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

71 of 177 references.

References · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2018Datasheets for Datasetscited by 1,386
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  11. 2018Model Cards for Model Reportingcited by 1,080
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
38 more
  1. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  2. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  4. 2017Deep Reinforcement Learning that Matterscited by 754
  5. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  6. 2019Green AIcited by 627
  7. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  8. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  9. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  10. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  11. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  12. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  13. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
  14. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  15. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  16. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  17. 2021The Values Encoded in Machine Learning Researchcited by 172
  18. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  19. 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 169
  20. 2022Compute Trends Across Three Eras of Machine Learningcited by 163
  21. 2020The Hardware Lotterycited by 160
  22. 2018Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningcited by 157
  23. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  24. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  25. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 145
  26. 2021Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithmscited by 144
  27. 2020With Little Power Comes Great Responsibilitycited by 142
  28. 2021The Benchmark Lotterycited by 136
  29. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
  30. 2020Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligencecited by 101
  31. 2019Evaluating the Underlying Gender Bias in Contextualized Word Embeddingscited by 94
  32. 2016Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysiscited by 93
  33. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 92
  34. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  35. 2021On Inductive Biases for Heterogeneous Treatment Effect Estimationcited by 75
  36. 2020We Need to Talk About Random Splitscited by 71
  37. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 71
  38. 2020Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLPcited by 50

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 1; 2026: 4.

10 papers.

  1. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  2. 2024WisPerMed at "Discharge Me!": Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC-IVcited by 6
  3. 2024Statistical Uncertainty in Word Embeddings: GloVe-Vcited by 5
  4. 2024Investigating the Impact of Randomness on Reproducibility in Computer Vision: A Study on Applications in Civil Engineering and Medicinecited by 2
  5. 2023Case-level Breast Cancer Prediction for Real Hospital Settingscited by 1
  6. 2025Beyond statistical significance: Quantifying uncertainty and statistical variability in multilingual and multitask NLP evaluationcited by 1
  7. 2026Indirect Question Answering in English, German and Bavarian: A Challenging Task for High- and Low-Resource Languages Alikecited by 1
  8. 2026Large Language Models as Annotators for Machine Translation Quality Estimationcited by 0
  9. 2026DEPART: DEcomposing PARiTy across Multilingual LLMscited by 0
  10. 2026SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Modelscited by 0