# Population Transformer: Learning Population-level Representations of Neural Activity

arXiv: https://arxiv.org/abs/2406.03044 · HTML: https://mlsanity.com/paper/2406.03044/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 58 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Graph-Guided Network for Irregularly Sampled Multivariate Time Series](https://mlsanity.com/paper/2110.05357/) (2021)
- [Brain Network Transformer](https://mlsanity.com/paper/2210.06681/) (2022)
- [MAEEG: Masked Auto-encoder for EEG Representation Learning](https://mlsanity.com/paper/2211.02625/) (2022)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data](https://mlsanity.com/paper/2206.11417/) (2022)
- [TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis](https://mlsanity.com/paper/2402.16412/) (2024)
- [MBrain: A Multi-channel Self-Supervised Learning Framework for Brain Signals](https://mlsanity.com/paper/2306.13102/) (2023)
- [STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer](https://mlsanity.com/paper/2206.04727/) (2022)
- [Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers](https://mlsanity.com/paper/2206.06131/) (2022)
- [Robust alignment of cross-session recordings of neural population activity by behaviour via unsupervised domain adaptation](https://mlsanity.com/paper/2202.06159/) (2022)
- [Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data](https://mlsanity.com/paper/2311.00136/) (2023)
- [Deep Neural Imputation: A Framework for Recovering Incomplete Brain Recordings](https://mlsanity.com/paper/2206.08094/) (2022)
- [Generalizability Under Sensor Failure: Tokenization + Transformers Enable More Robust Latent Spaces](https://mlsanity.com/paper/2402.18546/) (2024)

## Built on by (most cited first, 26 of 26)

- [LUNA: Efficient and Topology-Agnostic Foundation Model for EEG Signal Analysis](https://mlsanity.com/paper/2510.22257/) (2025)
- [Generalizable, real-time neural decoding with hybrid state-space models](https://mlsanity.com/paper/2506.05320/) (2025)
- [POCO: Scalable Neural Forecasting through Population Conditioning](https://mlsanity.com/paper/2506.14957/) (2025)
- [BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity](https://mlsanity.com/paper/2512.12135/) (2025)
- [DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representations](https://mlsanity.com/paper/2512.19097/) (2025)
- [Neuroprobe: Evaluating Intracranial Brain Responses to Naturalistic Stimuli](https://mlsanity.com/paper/2509.21671/) (2025)
- [Cross-Modal Representational Knowledge Distillation for Enhanced Spike-Informed LFP Modeling](https://mlsanity.com/paper/2512.12461/) (2025)
- [DIVER-0 : A Fully Channel Equivariant EEG Foundation Model](https://mlsanity.com/paper/2507.14141/) (2025)
- [Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG](https://mlsanity.com/paper/2602.23410/) (2026)
- [Unsupervised discovery of the shared and private geometry in multi-view data](https://mlsanity.com/paper/2408.12091/) (2024)
- [BrainStratify: Coarse-to-Fine Disentanglement of Intracranial Neural Dynamics](https://mlsanity.com/paper/2505.20480/) (2025)
- [Functional embeddings enable Aggregation of multi-area SEEG recordings over subjects and sessions](https://mlsanity.com/paper/2510.27090/) (2025)
- [Learning Time-Scale Invariant Population-Level Neural Representations](https://mlsanity.com/paper/2511.13022/) (2025)
- [From Minutes to Days: Scaling Intracranial Speech Decoding with Supervised Pretraining](https://mlsanity.com/paper/2512.15830/) (2025)
- [Cross-Subject Modeling for Widefield Calcium Imaging via Atlas-Aligned Spatiotemporal Tokenization](https://mlsanity.com/paper/2607.09754/) (2026)
- [NLP4Neuro: Sequence-to-sequence learning for neural population decoding](https://mlsanity.com/paper/2507.02264/) (2025)
- [Learning the relative composition of EEG signals using pairwise relative shift pretraining](https://mlsanity.com/paper/2511.11940/) (2025)
- [RPNT: Robust Pre-trained Neural Transformer -- A Pathway for Generalized Motor Decoding](https://mlsanity.com/paper/2601.17641/) (2026)
- [Self-Supervised Learning via Flow-Guided Neural Operator on Time-Series Data](https://mlsanity.com/paper/2602.12267/) (2026)
- [RECTOR: Masked Region-Channel-Temporal Modeling for Affective and Cognitive Representation Learning](https://mlsanity.com/paper/2606.15278/) (2026)
- [Device Passport: Enabling Spatio-Temporal Pretrained Models to Generalize Across Input Layouts](https://mlsanity.com/paper/2607.00249/) (2026)
- [iBrain: A Unified Foundation Model Reading the Brain from Surface to Spikes](https://mlsanity.com/paper/2609.06960/) (2026)
- [Pretraining for Sample-Efficient Neural Interfaces](https://mlsanity.com/paper/2609.13507/) (2026)
- [iMINDBench: iEEG Multi-Institution Neural Decoding Benchmark](https://mlsanity.com/paper/2609.18104/) (2026)
- [BAT-CLIP: Trimodal Alignment of Brain, Audio and Text](https://mlsanity.com/paper/2609.31180/) (2026)
- [PHASE: A Physiology-Guided Hierarchical Foundation Model for Intracranial EEG](https://mlsanity.com/paper/2609.36087/) (2026)

## Abstract (arXiv)

We present a self-supervised framework that learns population-level codes for arbitrary ensembles of neural recordings at scale. We address key challenges in scaling models with neural time-series data, namely, sparse and variable electrode distribution across subjects and datasets. The Population Transformer (PopT) stacks on top of pretrained temporal embeddings and enhances downstream decoding by enabling learned aggregation of multiple spatially-sparse data channels. The pretrained PopT lowers the amount of data required for downstream decoding experiments, while increasing accuracy, even on held-out subjects and tasks. Compared to end-to-end methods, this approach is computationally lightweight, while achieving similar or better decoding performance. We further show how our framework is generalizable to multiple time-series embeddings and neural data modalities. Beyond decoding, we interpret the pretrained and fine-tuned PopT models to show how they can be used to extract neuroscience insights from large amounts of data. We release our code as well as a pretrained PopT to enable off-the-shelf improvements in multi-channel intracranial data decoding and interpretability. Code is available at https://github.com/czlwang/PopulationTransformer.