# AI Knowledge and Reasoning: Emulating Expert Creativity in Scientific Research

arXiv: https://arxiv.org/abs/2404.04436 · HTML: https://mlsanity.com/paper/2404.04436/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 10 of 10)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [The Turing Trap: The Promise & Peril of Human-Like Artificial Intelligence](https://mlsanity.com/paper/2201.04200/) (2022)
- [Artificial Intelligence and Life in 2030: The One Hundred Year Study on Artificial Intelligence](https://mlsanity.com/paper/2211.06318/) (2022)

## Built on by (most cited first, 3 of 3)

- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology](https://mlsanity.com/paper/2602.16703/) (2026)
- [Rise of Generative Artificial Intelligence in Science](https://mlsanity.com/paper/2412.20960/) (2024)

## Abstract (arXiv)

We investigate whether modern AI can emulate expert creativity in complex scientific endeavors. We introduce novel methodology that utilizes original research articles published after the AI's training cutoff, ensuring no prior exposure, mitigating concerns of rote memorization and prior training. The AI are tasked with redacting findings, predicting outcomes from redacted research, and assessing prediction accuracy against reported results. Analysis on 589 published studies in four leading psychology journals over a 28-month period, showcase the AI's proficiency in understanding specialized research, deductive reasoning, and evaluating evidentiary alignment--cognitive hallmarks of human subject matter expertise and creativity. These findings suggest the potential of general-purpose AI to transform academia, with roles requiring knowledge-based creativity become increasingly susceptible to technological substitution.