mlsanityData to 7 Oct 2026

Preregistering NLP Research

arXiv 2103.06944 ↗First posted 11 Mar 2021cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This practice is increasingly common in medicine and psychology, but is rarely discussed in NLP. This paper discusses preregistration in more detail, explores how NLP researchers could preregister their work, and presents several preregistration questions for different kinds of studies. Finally, we argue in favour of registered reports, which could provide firmer grounds for slow science in NLP research. The goal of this paper is to elicit a discussion in the NLP community, which we hope to synthesise into a general NLP preregistration form in future research.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 8 of its 76 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

8 of 76 references.

References · 8

  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2018Datasheets for Datasetscited by 1,371
  3. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  4. 2018Model Cards for Model Reportingcited by 1,068
  5. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  6. 2018Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answeringcited by 23
  7. 2018Do RNNs learn human-like abstract word order preferences?cited by 10
  8. 2018Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysiscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 7; 2024: 1; 2025: 1; 2026: 11.

24 papers.

Sort:

By area:

  1. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  2. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 22
  3. 2021Underreporting of errors in NLG output, and what to do about itcited by 19
  4. 2022Experimental Standards for Deep Learning in Natural Language Processing Researchcited by 9
  5. 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
  6. 2025Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigorcited by 7
  7. 2023A Two-Sided Discussion of Preregistration of NLP Researchcited by 6
  8. 2023Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processingcited by 3
  9. 2026Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstentioncited by 2
  10. 2022Neural Data-to-Text Generation Based on Small Datasets: Comparing the Added Value of Two Semi-Supervised Learning Approaches on Top of a Large Language Modelcited by 1
  11. 2023On Degrees of Freedom in Defining and Testing Natural Language Understandingcited by 1
  12. 2024Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Studycited by 1
12 more
  1. 2026When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVRcited by 1
  2. 2026Does a Tool Result Carry More Authority Than Plain Text? Three Prospective Studies of False-Claim Adoption in a Synthetic Assignment Task with Claude Opus 5cited by 1
  3. 2026What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agentscited by 1
  4. 2023Evaluating NLG systems: A brief introductioncited by 0
  5. 2023Sentiment Analysis in Digital Spaces: An Overview of Reviewscited by 0
  6. 2026Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstrationcited by 0
  7. 2026Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialoguescited by 0
  8. 2026Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Auditcited by 0
  9. 2026Why Didn't It Check? Unsupported Final Claims and Their Repair in Two Tool-Equipped Language Modelscited by 0
  10. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0
  11. 2026Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpointscited by 0
  12. 2026Grounded Revision vs. Prior Injection: Probing Retrieval-Augmented Patent Claim Amendmentcited by 0