# Meta predictive learning model of languages in neural circuits

arXiv: https://arxiv.org/abs/2309.04106 · HTML: https://mlsanity.com/paper/2309.04106/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Predictive Coding: a Theoretical and Experimental Review](https://mlsanity.com/paper/2107.12979/) (2021)
- [Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?](https://mlsanity.com/paper/2202.09467/) (2022)
- [Large Language Models and the Reverse Turing Test](https://mlsanity.com/paper/2207.14382/) (2022)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [On the relationship between predictive coding and backpropagation](https://mlsanity.com/paper/2106.13082/) (2021)
- [Constrained Predictive Coding as a Biologically Plausible Model of the Cortical Hierarchy](https://mlsanity.com/paper/2210.15752/) (2022)
- [Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks](https://mlsanity.com/paper/2103.03725/) (2021)
- [Gated recurrent neural networks discover attention](https://mlsanity.com/paper/2309.01775/) (2023)
- [Eight challenges in developing theory of intelligence](https://mlsanity.com/paper/2306.11232/) (2023)
- [Learning credit assignment](https://mlsanity.com/paper/2001.03354/) (2020)
- [Role of zero synapses in unsupervised feature learning](https://mlsanity.com/paper/1703.07943/) (2017)
- [Energy--Information Trade-off Induces Continuous and Discontinuous Phase Transitions in Lateral Predictive Coding](https://mlsanity.com/paper/2302.11681/) (2023)
- [Ensemble perspective for understanding temporal credit assignment](https://mlsanity.com/paper/2102.03740/) (2021)

## Built on by (most cited first, 2 of 2)

- [Neural Langevin Machine: a local asymmetric learning rule can be creative](https://mlsanity.com/paper/2506.23546/) (2025)
- [Spin glass model of in-context learning](https://mlsanity.com/paper/2408.02288/) (2024)

## Abstract (arXiv)

Large language models based on self-attention mechanisms have achieved astonishing performances not only in natural language itself, but also in a variety of tasks of different nature. However, regarding processing language, our human brain may not operate using the same principle. Then, a debate is established on the connection between brain computation and artificial self-supervision adopted in large language models. One of most influential hypothesis in brain computation is the predictive coding framework, which proposes to minimize the prediction error by local learning. However, the role of predictive coding and the associated credit assignment in language processing remains unknown. Here, we propose a mean-field learning model within the predictive coding framework, assuming that the synaptic weight of each connection follows a spike and slab distribution, and only the distribution, rather than specific weights, is trained. This meta predictive learning is successfully validated on classifying handwritten digits where pixels are input to the network in sequence, and moreover on the toy and real language corpus. Our model reveals that most of the connections become deterministic after learning, while the output connections have a higher level of variability. The performance of the resulting network ensemble changes continuously with data load, further improving with more training data, in analogy with the emergent behavior of large language models. Therefore, our model provides a starting point to investigate the connection among brain computation, next-token prediction and general intelligence.