mlsanityData to 14 Oct 2026

Long-range and hierarchical language predictions in brains and algorithms

arXiv 2111.14232 ↗First posted 28 Nov 2021q-bio.NC cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Deep learning has recently made remarkable progress in natural language processing. Yet, the resulting algorithms remain far from competing with the language abilities of the human brain. Predictive coding theory offers a potential explanation to this discrepancy: while deep language algorithms are optimized to predict adjacent words, the human brain would be tuned to make long-range and hierarchical predictions. To test this hypothesis, we analyze the fMRI brain signals of 304 subjects each listening to 70min of short stories. After confirming that the activations of deep language algorithms linearly map onto those of the brain, we show that enhancing these models with long-range forecast representations improves their brain-mapping. The results further reveal a hierarchy of predictions in the brain, whereby the fronto-parietal cortices forecast more abstract and more distant representations than the temporal cortices. Overall, this study strengthens predictive coding theory and suggests a critical role of long-range and hierarchical predictions in natural language processing.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 26 of its 70 references that are in the map. First posted on arXiv 28 Nov 2021.

Builds on

26 of 70 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2014Going Deeper with Convolutionscited by 4,241
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
14 more
  1. 2018Hierarchical Neural Story Generationcited by 1,193
  2. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  3. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  4. 2017Challenges in Data-to-Document Generationcited by 226
  5. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  6. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  7. 2014Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathwaycited by 94
  8. 2019The emergence of number and syntax units in LSTM language modelscited by 94
  9. 2019Linguistic generalization and compositionality in modern artificial neural networkscited by 58
  10. 2020Word meaning in minds and machinescited by 51
  11. 2021Disentangling Syntax and Semantics in the Brain with Deep Networkscited by 40
  12. 2021Model-based analysis of brain activity reveals the hierarchy of language in 305 subjectscited by 10
  13. 2019Tracking Naturalistic Linguistic Predictions with Deep Neural Language Modelscited by 9
  14. 2021Can RNNs learn Recursive Nested Subject-Verb Agreements?cited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 1; 2025: 0; 2026: 1.

10 papers.

  1. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
  2. 2023Do Vision and Language Models Share Concepts? A Vector Space Alignment Studycited by 27
  3. 2023Mapping Brains with Language Models: A Surveycited by 8
  4. 2023The Multiscale Surface Vision Transformercited by 6
  5. 2024Metric Learning Encoding Models: A Multivariate Framework for Interpreting Neural Representationscited by 3
  6. 2022MEG-MASC: a high-quality magneto-encephalography dataset for evaluating natural speech processingcited by 2
  7. 2023INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformerscited by 2
  8. 2022Executive Function: A Contrastive Value Policy for Resampling and Relabeling Perceptions via Hindsight Summarization?cited by 1
  9. 2023Attention, not scale, drives human-AI alignment in multimodal language predictioncited by 1
  10. 2026Late Transformer Layers Recode Syntax Canonically: Evidence from Greek Scrambling and Cross-Layer Generalisationcited by 0