# What Do NLP Researchers Believe? Results of the NLP Community Metasurvey arXiv: https://arxiv.org/abs/2208.12852 ยท HTML: https://mlsanity.com/paper/2208.12852/ ## Key facts - Cited by 28 papers in the map (data through 14 Oct 2026). - Builds on 19 of its 65 references that are in the map. - First posted on arXiv 26 Aug 2022. ## Builds on (most cited first, 19 of 19) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016) - [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019) - [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019) - [Green AI](https://mlsanity.com/paper/1907.10597/) (2019) - [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021) - [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019) - [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021) - [The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink](https://mlsanity.com/paper/2204.05149/) (2022) - [Pathologies of Neural Models Make Interpretations Difficult](https://mlsanity.com/paper/1804.07781/) (2018) - [Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?](https://mlsanity.com/paper/2207.10551/) (2022) - [Do Transformer Modifications Transfer Across Implementations and Applications?](https://mlsanity.com/paper/2102.11972/) (2021) - [Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?](https://mlsanity.com/paper/2104.10809/) (2021) - [Examining Citations of Natural Language Processing Literature](https://mlsanity.com/paper/2005.00912/) (2020) - [Gender Gap in Natural Language Processing Research: Disparities in Authorship and Citations](https://mlsanity.com/paper/2005.00962/) (2020) - [The State of NLP Literature: A Diachronic Analysis of the ACL Anthology](https://mlsanity.com/paper/1911.03562/) (2019) ## Built on by (most cited first, 28 of 28) - [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022) - [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023) - [The Debate Over Understanding in AI's Large Language Models](https://mlsanity.com/paper/2210.13966/) (2022) - [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023) - [The Gradient of Generative AI Release: Methods and Considerations](https://mlsanity.com/paper/2302.04844/) (2023) - [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025) - [Benchmarks as Microscopes: A Call for Model Metrology](https://mlsanity.com/paper/2407.16711/) (2024) - [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023) - [Creating a Large Language Model of a Philosopher](https://mlsanity.com/paper/2302.01339/) (2023) - [We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields](https://mlsanity.com/paper/2310.14870/) (2023) - [Reproducibility in NLP: What Have We Learned from the Checklist?](https://mlsanity.com/paper/2306.09562/) (2023) - [On pitfalls (and advantages) of sophisticated large language models](https://mlsanity.com/paper/2303.17511/) (2023) - [Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models](https://mlsanity.com/paper/2503.06263/) (2025) - [On the Origins of Bias in NLP through the Lens of the Jim Code](https://mlsanity.com/paper/2305.09281/) (2023) - [The Two Word Test: A Semantic Benchmark for Large Language Models](https://mlsanity.com/paper/2306.04610/) (2023) - [To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing](https://mlsanity.com/paper/2310.07715/) (2023) - [On Tables with Numbers, with Numbers](https://mlsanity.com/paper/2408.06062/) (2024) - [Language Models Understand Us, Poorly](https://mlsanity.com/paper/2210.10684/) (2022) - [Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processing](https://mlsanity.com/paper/2304.08315/) (2023) - [Research Borderlands: Analysing Writing Across Research Cultures](https://mlsanity.com/paper/2506.00784/) (2025) - [NeurIPS should lead scientific consensus on AI policy](https://mlsanity.com/paper/2510.00075/) (2025) - [Collaboration or Corporate Capture? Quantifying NLP's Reliance on Industry Artifacts and Contributions](https://mlsanity.com/paper/2312.03912/) (2023) - [What We Talk About When We Talk About LMs: Implicit Paradigm Shifts and the Ship of Language Models](https://mlsanity.com/paper/2407.01929/) (2024) - [Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey](https://mlsanity.com/paper/2503.09701/) (2025) - [Research Community Perspectives on "Intelligence" and Large Language Models](https://mlsanity.com/paper/2505.20959/) (2025) - [Can Large Language Models Create New Knowledge for Spatial Reasoning Tasks?](https://mlsanity.com/paper/2405.14379/) (2024) - [Good Intentions Beyond ACL: Who Does NLP for Social Good, and Where?](https://mlsanity.com/paper/2510.04434/) (2025) - [The Future of NLP may not be at NLP Conferences: Scholarly Migration Patterns in Natural Language Processing](https://mlsanity.com/paper/2607.02416/) (2026) ## Abstract (arXiv) We present the results of the NLP Community Metasurvey. Run from May to June 2022, the survey elicited opinions on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies: For example, respondents are split almost exactly in half on questions about the importance of artificial general intelligence, whether language models understand language, and the necessity of linguistic structure and inductive bias for solving NLP problems. In addition, the survey posed meta-questions, asking respondents to predict the distribution of survey responses. This allows us not only to gain insight on the spectrum of beliefs held by NLP researchers, but also to uncover false sociological beliefs where the community's predictions don't match reality. We find such mismatches on a wide range of issues. Among other results, the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its own belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.