# Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers

arXiv: https://arxiv.org/abs/2206.06131 · HTML: https://mlsanity.com/paper/2206.06131/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 80 references that are in the map.
- First posted on arXiv 10 Jun 2022.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Spatial-Temporal Transformer Networks for Traffic Flow Forecasting](https://mlsanity.com/paper/2001.02908/) (2020)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Spatial Temporal Transformer Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2012.06399/) (2020)
- [Dynamic Memory Networks for Visual and Textual Question Answering](https://mlsanity.com/paper/1603.01417/) (2016)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [GroupFormer: Group Activity Recognition with Clustered Spatial-Temporal Transformer](https://mlsanity.com/paper/2108.12630/) (2021)
- [Representation learning for neural population activity with Neural Data Transformers](https://mlsanity.com/paper/2108.01210/) (2021)
- [Exploring Interpretable LSTM Neural Networks over Multi-Variable Data](https://mlsanity.com/paper/1905.12034/) (2019)
- [Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction](https://mlsanity.com/paper/2102.10106/) (2021)

## Built on by (most cited first, 18 of 18)

- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution](https://mlsanity.com/paper/2407.14668/) (2024)
- [Population Transformer: Learning Population-level Representations of Neural Activity](https://mlsanity.com/paper/2406.03044/) (2024)
- [Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals](https://mlsanity.com/paper/2309.05927/) (2023)
- [Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models](https://mlsanity.com/paper/2410.09614/) (2024)
- [Learning Time-Invariant Representations for Individual Neurons from Population Dynamics](https://mlsanity.com/paper/2311.02258/) (2023)
- [SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding](https://mlsanity.com/paper/2507.08402/) (2025)
- [Extraction and Recovery of Spatio-Temporal Structure in Latent Dynamics Alignment with Diffusion Models](https://mlsanity.com/paper/2306.06138/) (2023)
- [SiBBlInGS: Similarity-driven Building-Block Inference using Graphs across States](https://mlsanity.com/paper/2306.04817/) (2023)
- [BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillation](https://mlsanity.com/paper/2410.13872/) (2024)
- [Cross-Modal Representational Knowledge Distillation for Enhanced Spike-Informed LFP Modeling](https://mlsanity.com/paper/2512.12461/) (2025)
- [Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance](https://mlsanity.com/paper/2402.11742/) (2024)
- [Know Thyself by Knowing Others: Learning Neuron Identity from Population Context](https://mlsanity.com/paper/2512.01199/) (2025)
- [Unsupervised discovery of the shared and private geometry in multi-view data](https://mlsanity.com/paper/2408.12091/) (2024)
- [Generalizable autoregressive modeling of time series through functional narratives](https://mlsanity.com/paper/2410.08421/) (2024)
- [Learning Time-Scale Invariant Population-Level Neural Representations](https://mlsanity.com/paper/2511.13022/) (2025)
- [Learning the relative composition of EEG signals using pairwise relative shift pretraining](https://mlsanity.com/paper/2511.11940/) (2025)
- [CalM: A Self-Supervised Foundation Model for Population Dynamics in Calcium Imaging Data](https://mlsanity.com/paper/2604.04958/) (2026)

## Abstract (arXiv)

Complex time-varying systems are often studied by abstracting away from the dynamics of individual components to build a model of the population-level dynamics from the start. However, when building a population-level description, it can be easy to lose sight of each individual and how they contribute to the larger picture. In this paper, we present a novel transformer architecture for learning from time-varying data that builds descriptions of both the individual as well as the collective population dynamics. Rather than combining all of our data into our model at the onset, we develop a separable architecture that operates on individual time-series first before passing them forward; this induces a permutation-invariance property and can be used to transfer across systems of different size and order. After demonstrating that our model can be applied to successfully recover complex interactions and dynamics in many-body systems, we apply our approach to populations of neurons in the nervous system. On neural activity datasets, we show that our model not only yields robust decoding performance, but also provides impressive performance in transfer across recordings of different animals without any neuron-level correspondence. By enabling flexible pre-training that can be transferred to neural recordings of different size and order, our work provides a first step towards creating a foundation model for neural decoding.