# Development of Cognitive Intelligence in Pre-trained Language Models

arXiv: https://arxiv.org/abs/2407.01047 · HTML: https://mlsanity.com/paper/2407.01047/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 86 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [In-Context Impersonation Reveals Large Language Models' Strengths and Biases](https://mlsanity.com/paper/2305.14930/) (2023)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [LLM360: Towards Fully Transparent Open-Source LLMs](https://mlsanity.com/paper/2312.06550/) (2023)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Stratified Rule-Aware Network for Abstract Visual Reasoning](https://mlsanity.com/paper/2002.06838/) (2020)
- [Do Language Embeddings Capture Scales?](https://mlsanity.com/paper/2010.05345/) (2020)
- [Word meaning in minds and machines](https://mlsanity.com/paper/2008.01766/) (2020)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Position: AI Evaluation Should Learn from How We Test Humans](https://mlsanity.com/paper/2306.10512/) (2023)
- [Do language models learn typicality judgments from text?](https://mlsanity.com/paper/2105.02987/) (2021)
- [Generative AI vs. AGI: The Cognitive Strengths and Weaknesses of Modern LLMs](https://mlsanity.com/paper/2309.10371/) (2023)
- [Comparing Machines and Children: Using Developmental Psychology Experiments to Assess the Strengths and Weaknesses of LaMDA Responses](https://mlsanity.com/paper/2305.11243/) (2023)
- [How Well Do Deep Learning Models Capture Human Concepts? The Case of the Typicality Effect](https://mlsanity.com/paper/2405.16128/) (2024)

## Built on by (most cited first, 3 of 3)

- [The Representational Geometry of Number](https://mlsanity.com/paper/2602.06843/) (2026)
- [LEVANTE-bench: Multi-Scale Comparison of VLMs to Children Using Cognitive Tasks (or, "Is Your VLM Smarter Than a 5th Grader?")](https://mlsanity.com/paper/2606.05497/) (2026)
- [On the use of foundation models in cognitive science](https://mlsanity.com/paper/2608.07812/) (2026)

## Abstract (arXiv)

Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). The increasing cognitive alignment of these models has made them candidates for cognitive science theories. Prior research into the emergent cognitive abilities of PLMs has largely been path-independent to model training, i.e., has focused on the final model weights and not the intermediate steps. However, building plausible models of human cognition using PLMs would benefit from considering the developmental alignment of their performance during training to the trajectories of children's thinking. Guided by psychometric tests of human intelligence, we choose four sets of tasks to investigate the alignment of ten popular families of PLMs and evaluate their available intermediate and final training steps. These tasks are Numerical ability, Linguistic abilities, Conceptual understanding, and Fluid reasoning. We find a striking regularity: regardless of model size, the developmental trajectories of PLMs consistently exhibit a window of maximal alignment to human cognitive development. Before that window, training appears to endow "blank slate" models with the requisite structure to be poised to rapidly learn from experience. After that window, training appears to serve the engineering goal of reducing loss but not the scientific goal of increasing alignment with human cognition.