# Preregistering NLP Research

arXiv: https://arxiv.org/abs/2103.06944 · HTML: https://mlsanity.com/paper/2103.06944/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 76 references that are in the map.
- First posted on arXiv 11 Mar 2021.

## Builds on (most cited first, 8 of 8)

- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering](https://mlsanity.com/paper/1806.04330/) (2018)
- [Do RNNs learn human-like abstract word order preferences?](https://mlsanity.com/paper/1811.01866/) (2018)
- [Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis](https://mlsanity.com/paper/1806.05219/) (2018)

## Built on by (most cited first, 25 of 25)

- [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021)
- [Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement](https://mlsanity.com/paper/2301.10684/) (2023)
- [Underreporting of errors in NLG output, and what to do about it](https://mlsanity.com/paper/2108.01182/) (2021)
- [Experimental Standards for Deep Learning in Natural Language Processing Research](https://mlsanity.com/paper/2204.06251/) (2022)
- [Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://mlsanity.com/paper/2312.13871/) (2023)
- [Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor](https://mlsanity.com/paper/2506.14652/) (2025)
- [A Two-Sided Discussion of Preregistration of NLP Research](https://mlsanity.com/paper/2302.10086/) (2023)
- [When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR](https://mlsanity.com/paper/2607.11022/) (2026)
- [Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention](https://mlsanity.com/paper/2608.13959/) (2026)
- [Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processing](https://mlsanity.com/paper/2304.08315/) (2023)
- [What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents](https://mlsanity.com/paper/2609.04518/) (2026)
- [Neural Data-to-Text Generation Based on Small Datasets: Comparing the Added Value of Two Semi-Supervised Learning Approaches on Top of a Large Language Model](https://mlsanity.com/paper/2207.06839/) (2022)
- [On Degrees of Freedom in Defining and Testing Natural Language Understanding](https://mlsanity.com/paper/2305.15130/) (2023)
- [Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Study](https://mlsanity.com/paper/2404.03987/) (2024)
- [Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstration](https://mlsanity.com/paper/2607.14399/) (2026)
- [Does a Tool Result Carry More Authority Than Plain Text? Three Prospective Studies of False-Claim Adoption in a Synthetic Assignment Task with Claude Opus 5](https://mlsanity.com/paper/2608.14992/) (2026)
- [Evaluating NLG systems: A brief introduction](https://mlsanity.com/paper/2303.16742/) (2023)
- [Sentiment Analysis in Digital Spaces: An Overview of Reviews](https://mlsanity.com/paper/2310.19687/) (2023)
- [Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues](https://mlsanity.com/paper/2608.12599/) (2026)
- [Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit](https://mlsanity.com/paper/2608.27309/) (2026)
- [Why Didn't It Check? Unsupported Final Claims and Their Repair in Two Tool-Equipped Language Models](https://mlsanity.com/paper/2608.27768/) (2026)
- [No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus](https://mlsanity.com/paper/2608.28875/) (2026)
- [Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints](https://mlsanity.com/paper/2609.04198/) (2026)
- [Grounded Revision vs. Prior Injection: Probing Retrieval-Augmented Patent Claim Amendment](https://mlsanity.com/paper/2609.36550/) (2026)
- [Calibrated Evidence Inference With a Small, Permissively Licensed Open Model](https://mlsanity.com/paper/2610.07019/) (2026)

## Abstract (arXiv)

Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This practice is increasingly common in medicine and psychology, but is rarely discussed in NLP. This paper discusses preregistration in more detail, explores how NLP researchers could preregister their work, and presents several preregistration questions for different kinds of studies. Finally, we argue in favour of registered reports, which could provide firmer grounds for slow science in NLP research. The goal of this paper is to elicit a discussion in the NLP community, which we hope to synthesise into a general NLP preregistration form in future research.