# Research Community Perspectives on "Intelligence" and Large Language Models

arXiv: https://arxiv.org/abs/2505.20959 · HTML: https://mlsanity.com/paper/2505.20959/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 61 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 22 of 22)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [HEAD-QA: A Healthcare Dataset for Complex Reasoning](https://mlsanity.com/paper/1906.04701/) (2019)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Competency Problems: On Finding and Removing Artifacts in Language Data](https://mlsanity.com/paper/2104.08646/) (2021)
- [Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?](https://mlsanity.com/paper/2104.10809/) (2021)
- [Targeting the Benchmark: On Methodology in Current Natural Language Processing Research](https://mlsanity.com/paper/2007.04792/) (2020)
- [Why AI is Harder Than We Think](https://mlsanity.com/paper/2104.12871/) (2021)
- [What Do NLP Researchers Believe? Results of the NLP Community Metasurvey](https://mlsanity.com/paper/2208.12852/) (2022)
- [Do Vision and Language Models Share Concepts? A Vector Space Alignment Study](https://mlsanity.com/paper/2302.06555/) (2023)
- [Machine Reading, Fast and Slow: When Do Models "Understand" Language?](https://mlsanity.com/paper/2209.07430/) (2022)

## Built on by (most cited first, 2 of 2)

- [On the Notion that Language Models Reason](https://mlsanity.com/paper/2511.11810/) (2025)
- [Like a Therapist, But Not: Reddit Narratives of AI in Mental Health Contexts](https://mlsanity.com/paper/2601.20747/) (2026)

## Abstract (arXiv)

Despite the widespread use of ''artificial intelligence'' (AI) framing in Natural Language Processing (NLP) research, it is not clear what researchers mean by ''intelligence''. To that end, we present the results of a survey on the notion of ''intelligence'' among researchers and its role in the research agenda. The survey elicited complete responses from 303 researchers from a variety of fields including NLP, Machine Learning (ML), Cognitive Science, Linguistics, and Neuroscience. We identify 3 criteria of intelligence that the community agrees on the most: generalization, adaptability, & reasoning. Our results suggests that the perception of the current NLP systems as ''intelligent'' is a minority position (29%). Furthermore, only 16.2% of the respondents see developing intelligent systems as a research goal, and these respondents are more likely to consider the current systems intelligent.