# Neural Encoding and Decoding at Scale

arXiv: https://arxiv.org/abs/2504.08201 · HTML: https://mlsanity.com/paper/2504.08201/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 64 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [4M: Massively Multimodal Masked Modeling](https://mlsanity.com/paper/2312.06647/) (2023)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision](https://mlsanity.com/paper/1608.03425/) (2016)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Machine learning for neural decoding](https://mlsanity.com/paper/1708.00909/) (2017)
- [Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution](https://mlsanity.com/paper/2407.14668/) (2024)
- [Targeted Neural Dynamical Modeling](https://mlsanity.com/paper/2110.14853/) (2021)
- [Promoting cross-modal representations to improve multimodal foundation models for physiological signals](https://mlsanity.com/paper/2410.16424/) (2024)
- [Building population models for large-scale neural recordings: opportunities and pitfalls](https://mlsanity.com/paper/2102.01807/) (2021)
- [Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data](https://mlsanity.com/paper/2310.03111/) (2023)

## Built on by (most cited first, 20 of 20)

- [PosterGen: Aesthetic-Aware Multi-Modal Paper-to-Poster Generation via Multi-Agent LLMs](https://mlsanity.com/paper/2508.17188/) (2025)
- [Generalizable, real-time neural decoding with hybrid state-space models](https://mlsanity.com/paper/2506.05320/) (2025)
- [SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding](https://mlsanity.com/paper/2507.08402/) (2025)
- [Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining](https://mlsanity.com/paper/2507.09513/) (2025)
- [Inpainting the Neural Picture: Inferring Unrecorded Brain Area Dynamics from Multi-Animal Datasets](https://mlsanity.com/paper/2510.11924/) (2025)
- [A cross-species neural foundation model for end-to-end speech decoding](https://mlsanity.com/paper/2511.21740/) (2025)
- [OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens](https://mlsanity.com/paper/2604.18827/) (2026)
- [Know Thyself by Knowing Others: Learning Neuron Identity from Population Context](https://mlsanity.com/paper/2512.01199/) (2025)
- [Cross-Subject Modeling for Widefield Calcium Imaging via Atlas-Aligned Spatiotemporal Tokenization](https://mlsanity.com/paper/2607.09754/) (2026)
- [Leveraging unlabelled data for generalizable neural population decoding](https://mlsanity.com/paper/2607.14086/) (2026)
- [RPNT: Robust Pre-trained Neural Transformer -- A Pathway for Generalized Motor Decoding](https://mlsanity.com/paper/2601.17641/) (2026)
- [CalM: A Self-Supervised Foundation Model for Population Dynamics in Calcium Imaging Data](https://mlsanity.com/paper/2604.04958/) (2026)
- [Integrative neurocybernetic modeling in the era of large-scale neuroscience](https://mlsanity.com/paper/2604.23903/) (2026)
- [Machine Learning Methods for Studying Latent Neural Activity Dynamics](https://mlsanity.com/paper/2606.10530/) (2026)
- [Real-time Reconstruction of Human Visual Perception from fMRI](https://mlsanity.com/paper/2607.22753/) (2026)
- [CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics](https://mlsanity.com/paper/2607.23258/) (2026)
- [NeuroPB: Scaling Neural Decoding with Pretrained Behavioral Representations](https://mlsanity.com/paper/2608.04389/) (2026)
- [Brain-Conditioned Action Policies for Neural Motor Decoding](https://mlsanity.com/paper/2609.34561/) (2026)
- [NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings](https://mlsanity.com/paper/2610.02864/) (2026)
- [Neural Data Needs Semantic Tokenization: Behavioral Events as Boundaries of Session-Transferable Tokens](https://mlsanity.com/paper/2610.03001/) (2026)

## Abstract (arXiv)

Recent work has demonstrated that large-scale, multi-animal models are powerful tools for characterizing the relationship between neural activity and behavior. Current large-scale approaches, however, focus exclusively on either predicting neural activity from behavior (encoding) or predicting behavior from neural activity (decoding), limiting their ability to capture the bidirectional relationship between neural activity and behavior. To bridge this gap, we introduce a multimodal, multi-task model that enables simultaneous Neural Encoding and Decoding at Scale (NEDS). Central to our approach is a novel multi-task-masking strategy, which alternates between neural, behavioral, within-modality, and cross-modality masking. We pretrain our method on the International Brain Laboratory (IBL) repeated site dataset, which includes recordings from 83 animals performing the same visual decision-making task. In comparison to other large-scale models, we demonstrate that NEDS achieves state-of-the-art performance for both encoding and decoding when pretrained on multi-animal data and then fine-tuned on new animals. Surprisingly, NEDS's learned embeddings exhibit emergent properties: even without explicit training, they are highly predictive of the brain regions in each recording. Altogether, our approach is a step towards a foundation model of the brain that enables seamless translation between neural activity and behavior.