mlsanityData to 13 Oct 2026

Syntactic Learnability of Echo State Neural Language Models at Scale

arXiv 2503.01724 ↗First posted 3 Mar 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

What is a neural model with minimum architectural complexity that exhibits reasonable language learning capability? To explore such a simple but sufficient neural language model, we revisit a basic reservoir computing (RC) model, Echo State Network (ESN), a restricted class of simple Recurrent Neural Networks. Our experiments showed that ESN with a large hidden state is comparable or superior to Transformer in grammaticality judgment tasks when trained with about 100M words, suggesting that architectures as complex as that of Transformer may not always be necessary for syntactic learning.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

17 of 31 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2020Scaling Laws for Neural Language Modelscited by 6,494
  4. 2016Layer Normalizationcited by 5,624
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,601
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,488
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,391
  8. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,049
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 557
  11. 2018Colorless green recurrent networks dream hierarchicallycited by 211
  12. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 118
5 more
  1. 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
  2. 2019No Training Required: Exploring Random Encoders for Sentence Classificationcited by 32
  3. 2024[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 26
  4. 2021Reservoir Computing with Diverse Timescales for Prediction of Multiscale Dynamicscited by 13
  5. 2020Sparsity in Reservoir Computing Neural Networkscited by 6

Built on by

Not cited yet.