# Single layer tiny Co4 outpaces GPT-2 and GPT-BERT

arXiv: https://arxiv.org/abs/2510.08404 · HTML: https://mlsanity.com/paper/2510.08404/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 74 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [The Computational Limits of Deep Learning](https://mlsanity.com/paper/2007.05558/) (2020)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models](https://mlsanity.com/paper/2405.09605/) (2024)
- [Local plasticity rules can learn deep representations using self-supervised contrastive predictions](https://mlsanity.com/paper/2010.08262/) (2020)
- [Single-phase deep learning in cortico-cortical networks](https://mlsanity.com/paper/2206.11769/) (2022)
- [The Dynamics of Functional Brain Networks: Integrated Network States during Cognitive Function](https://mlsanity.com/paper/1511.02976/) (2015)
- [A comparison of partial information decompositions using data from real and simulated layer 5b pyramidal cells](https://mlsanity.com/paper/2206.06456/) (2022)
- [Unlocking the potential of two-point cells for energy-efficient and resilient training of deep nets](https://mlsanity.com/paper/2211.01950/) (2022)
- [Two-compartment neuronal spiking model expressing brain-state specific apical-amplification, -isolation and -drive regimes](https://mlsanity.com/paper/2311.06074/) (2023)
- [An Overlooked Role of Context-Sensitive Dendrites](https://mlsanity.com/paper/2408.11019/) (2024)
- [Beyond Attention: Toward Machines with Intrinsic Higher Mental States](https://mlsanity.com/paper/2505.06257/) (2025)

## Abstract (arXiv)

We show that a tiny Co4 machine(Adeel,2025) with a single layer, two heads, and 8M parameters, operating at an approximate cost of O(N) (where N is the number of input tokens), outpaces the BabyLM Challenge baselines GPT-2 (124M, 12 layers, O(N2)) and GPT-BERT (30M, 12 layers, O(N2)) in just two epochs, while both are trained for ten. Co4 achieves orders-of-magnitude greater training efficiency on 10M tokens, demonstrating highly sample efficient pretraining. Using the BabyLM challenge evaluation pipeline across complex benchmarks, Co4 exhibits strong zero-shot and fine-tuning performance on SuperGLUE tasks. Specifically, Co4 outperforms GPT-2 on 5 out of 7 zero-shot metrics and 6 out of 7 fine-tuning tasks, and GPT-BERT on 4 out of 7 metrics in both cases. These results suggest the need to rethink prevailing deep learning paradigms and associated scaling laws.