mlsanityData to 14 Oct 2026

Research Community Perspectives on "Intelligence" and Large Language Models

arXiv 2505.20959 ↗First posted 27 May 2025cs.CL cs.CYcited by 2

Not on the map. Why?

Abstract (arXiv)

Despite the widespread use of ''artificial intelligence'' (AI) framing in Natural Language Processing (NLP) research, it is not clear what researchers mean by ''intelligence''. To that end, we present the results of a survey on the notion of ''intelligence'' among researchers and its role in the research agenda. The survey elicited complete responses from 303 researchers from a variety of fields including NLP, Machine Learning (ML), Cognitive Science, Linguistics, and Neuroscience. We identify 3 criteria of intelligence that the community agrees on the most: generalization, adaptability, & reasoning. Our results suggests that the perception of the current NLP systems as ''intelligent'' is a minority position (29%). Furthermore, only 16.2% of the respondents see developing intelligent systems as a research goal, and these respondents are more likely to consider the current systems intelligent.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 61 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

22 of 61 references.

References · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  7. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  9. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  11. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  12. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
10 more
  1. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
  2. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 101
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  5. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  6. 2020Targeting the Benchmark: On Methodology in Current Natural Language Processing Researchcited by 45
  7. 2021Why AI is Harder Than We Thinkcited by 41
  8. 2022What Do NLP Researchers Believe? Results of the NLP Community Metasurveycited by 28
  9. 2023Do Vision and Language Models Share Concepts? A Vector Space Alignment Studycited by 27
  10. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025On the Notion that Language Models Reasoncited by 2
  2. 2026Like a Therapist, But Not: Reddit Narratives of AI in Mental Health Contextscited by 2