# Deep Recurrent Encoder: A scalable end-to-end network to model brain signals

arXiv: https://arxiv.org/abs/2103.02339 · HTML: https://mlsanity.com/paper/2103.02339/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 105 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Deep learning-based electroencephalography analysis: a systematic review](https://mlsanity.com/paper/1901.05498/) (2019)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Towards an integration of deep learning and neuroscience](https://mlsanity.com/paper/1606.03813/) (2016)
- [A Clockwork RNN](https://mlsanity.com/paper/1402.3511/) (2014)
- [Universality and individuality in neural dynamics across large populations of recurrent networks](https://mlsanity.com/paper/1907.08549/) (2019)
- [Learning interpretable continuous-time models of latent stochastic dynamical systems](https://mlsanity.com/paper/1902.04420/) (2019)
- [Neural system identification for large populations separating "what" and "where"](https://mlsanity.com/paper/1711.02653/) (2017)
- [SING: Symbol-to-Instrument Neural Generator](https://mlsanity.com/paper/1810.09785/) (2018)
- [Modeling Shared Responses in Neuroimaging Studies through MultiView ICA](https://mlsanity.com/paper/2006.06635/) (2020)
- [Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech](https://mlsanity.com/paper/2103.01032/) (2021)
- [Multi-timescale Representation Learning in LSTM Language Models](https://mlsanity.com/paper/2009.12727/) (2020)
- [Modeling the dynamics of human brain activity with recurrent neural networks](https://mlsanity.com/paper/1606.03071/) (2016)
- [Neural network models and deep learning - a primer for biologists](https://mlsanity.com/paper/1902.04704/) (2019)
- [Dynamic Decomposition of Spatiotemporal Neural Signals](https://mlsanity.com/paper/1605.02609/) (2016)
- [Deep adversarial neural decoding](https://mlsanity.com/paper/1705.07109/) (2017)

## Built on by (most cited first, 6 of 6)

- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Scaling laws for decoding images from brain activity](https://mlsanity.com/paper/2501.15322/) (2025)
- [CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding](https://mlsanity.com/paper/2402.08994/) (2024)
- [GPT2MEG: Quantizing MEG for Autoregressive Generation](https://mlsanity.com/paper/2404.09256/) (2024)
- [MEG-MASC: a high-quality magneto-encephalography dataset for evaluating natural speech processing](https://mlsanity.com/paper/2208.11488/) (2022)
- [FedMinds: Privacy-Preserving Personalized Brain Visual Decoding](https://mlsanity.com/paper/2409.02044/) (2024)

## Abstract (arXiv)

Understanding how the brain responds to sensory inputs is challenging: brain recordings are partial, noisy, and high dimensional; they vary across sessions and subjects and they capture highly nonlinear dynamics. These challenges have led the community to develop a variety of preprocessing and analytical (almost exclusively linear) methods, each designed to tackle one of these issues. Instead, we propose to address these challenges through a specific end-to-end deep learning architecture, trained to predict the brain responses of multiple subjects at once. We successfully test this approach on a large cohort of magnetoencephalography (MEG) recordings acquired during a one-hour reading task. Our Deep Recurrent Encoding (DRE) architecture reliably predicts MEG responses to words with a three-fold improvement over classic linear methods. To overcome the notorious issue of interpretability of deep learning, we describe a simple variable importance analysis. When applied to DRE, this method recovers the expected evoked responses to word length and word frequency. The quantitative improvement of the present deep learning approach paves the way to better understand the nonlinear dynamics of brain activity from large datasets.