# Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution

arXiv: https://arxiv.org/abs/2407.14668 · HTML: https://mlsanity.com/paper/2407.14668/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Representation learning for neural population activity with Neural Data Transformers](https://mlsanity.com/paper/2108.01210/) (2021)
- [Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data](https://mlsanity.com/paper/2206.11417/) (2022)
- [Large-scale neural recordings call for new insights to link brain and behavior](https://mlsanity.com/paper/2103.14662/) (2021)
- [Enabling hyperparameter optimization in sequential autoencoders for spiking neural data](https://mlsanity.com/paper/1908.07896/) (2019)
- [Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers](https://mlsanity.com/paper/2206.06131/) (2022)
- [Do Brain Networks Evolve by Maximizing their Information Flow Capacity?](https://mlsanity.com/paper/1508.03527/) (2015)

## Built on by (most cited first, 28 of 28)

- [Neural Encoding and Decoding at Scale](https://mlsanity.com/paper/2504.08201/) (2025)
- [Generalizable, real-time neural decoding with hybrid state-space models](https://mlsanity.com/paper/2506.05320/) (2025)
- [SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding](https://mlsanity.com/paper/2507.08402/) (2025)
- [POCO: Scalable Neural Forecasting through Population Conditioning](https://mlsanity.com/paper/2506.14957/) (2025)
- [BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity](https://mlsanity.com/paper/2512.12135/) (2025)
- [Adaptive Intelligence: leveraging insights from adaptive behavior in animals to build flexible AI systems](https://mlsanity.com/paper/2411.15234/) (2024)
- [Scaling Vision Transformers for Functional MRI with Flat Maps](https://mlsanity.com/paper/2510.13768/) (2025)
- [Inpainting the Neural Picture: Inferring Unrecorded Brain Area Dynamics from Multi-Animal Datasets](https://mlsanity.com/paper/2510.11924/) (2025)
- [A cross-species neural foundation model for end-to-end speech decoding](https://mlsanity.com/paper/2511.21740/) (2025)
- [Neuroprobe: Evaluating Intracranial Brain Responses to Naturalistic Stimuli](https://mlsanity.com/paper/2509.21671/) (2025)
- [Cross-Modal Representational Knowledge Distillation for Enhanced Spike-Informed LFP Modeling](https://mlsanity.com/paper/2512.12461/) (2025)
- [Benchmarking Probabilistic Time Series Forecasting Models on Neural Activity](https://mlsanity.com/paper/2510.18037/) (2025)
- [Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse](https://mlsanity.com/paper/2605.12808/) (2026)
- [Cross-Subject Modeling for Widefield Calcium Imaging via Atlas-Aligned Spatiotemporal Tokenization](https://mlsanity.com/paper/2607.09754/) (2026)
- [Leveraging unlabelled data for generalizable neural population decoding](https://mlsanity.com/paper/2607.14086/) (2026)
- [NS-Copilot: An LLM-Driven Agent System for Autonomous Neuroscience Analysis](https://mlsanity.com/paper/2609.01971/) (2026)
- [NLP4Neuro: Sequence-to-sequence learning for neural population decoding](https://mlsanity.com/paper/2507.02264/) (2025)
- [Latent Graph Learning in Generative Models of Neural Signals](https://mlsanity.com/paper/2508.16776/) (2025)
- [RPNT: Robust Pre-trained Neural Transformer -- A Pathway for Generalized Motor Decoding](https://mlsanity.com/paper/2601.17641/) (2026)
- [CalM: A Self-Supervised Foundation Model for Population Dynamics in Calcium Imaging Data](https://mlsanity.com/paper/2604.04958/) (2026)
- [REALM: Retrospective Encoder Alignment for LFP Modeling](https://mlsanity.com/paper/2605.14867/) (2026)
- [Group-Aware Matrix Estimation and Latent Subspace Recovery](https://mlsanity.com/paper/2605.20559/) (2026)
- [Machine Learning Methods for Studying Latent Neural Activity Dynamics](https://mlsanity.com/paper/2606.10530/) (2026)
- [Masked Autoencoders Learn Perception-Relevant Representations from Resting State Neural Data](https://mlsanity.com/paper/2607.22615/) (2026)
- [CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics](https://mlsanity.com/paper/2607.23258/) (2026)
- [iBrain: A Unified Foundation Model Reading the Brain from Surface to Spikes](https://mlsanity.com/paper/2609.06960/) (2026)
- [Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces](https://mlsanity.com/paper/2609.27441/) (2026)
- [NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings](https://mlsanity.com/paper/2610.02864/) (2026)

## Abstract (arXiv)

Neuroscience research has made immense progress over the last decade, but our understanding of the brain remains fragmented and piecemeal: the dream of probing an arbitrary brain region and automatically reading out the information encoded in its neural activity remains out of reach. In this work, we build towards a first foundation model for neural spiking data that can solve a diverse set of tasks across multiple brain areas. We introduce a novel self-supervised modeling approach for population activity in which the model alternates between masking out and reconstructing neural activity across different time steps, neurons, and brain regions. To evaluate our approach, we design unsupervised and supervised prediction tasks using the International Brain Laboratory repeated site dataset, which is comprised of Neuropixels recordings targeting the same brain locations across 48 animals and experimental sessions. The prediction tasks include single-neuron and region-level activity prediction, forward prediction, and behavior decoding. We demonstrate that our multi-task-masking (MtM) approach significantly improves the performance of current state-of-the-art population models and enables multi-task learning. We also show that by training on multiple animals, we can improve the generalization ability of the model to unseen animals, paving the way for a foundation model of the brain at single-cell, single-spike resolution.