# NLP4Neuro: Sequence-to-sequence learning for neural population decoding

arXiv: https://arxiv.org/abs/2507.02264 · HTML: https://mlsanity.com/paper/2507.02264/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Next Generation Reservoir Computing](https://mlsanity.com/paper/2106.07688/) (2021)
- [BrainBERT: Self-supervised representation learning for intracranial recordings](https://mlsanity.com/paper/2302.14367/) (2023)
- [Learning Natural Language Inference with LSTM](https://mlsanity.com/paper/1512.08849/) (2015)
- [Extraction of Salient Sentences from Labelled Documents](https://mlsanity.com/paper/1412.6815/) (2014)
- [Representation learning for neural population activity with Neural Data Transformers](https://mlsanity.com/paper/2108.01210/) (2021)
- [Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study](https://mlsanity.com/paper/2307.08072/) (2023)
- [MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problems](https://mlsanity.com/paper/2306.17525/) (2023)
- [Large-scale neural recordings call for new insights to link brain and behavior](https://mlsanity.com/paper/2103.14662/) (2021)
- [Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution](https://mlsanity.com/paper/2407.14668/) (2024)
- [Population Transformer: Learning Population-level Representations of Neural Activity](https://mlsanity.com/paper/2406.03044/) (2024)
- [MLPST: MLP is All You Need for Spatio-Temporal Prediction](https://mlsanity.com/paper/2309.13363/) (2023)
- [ZAPBench: A Benchmark for Whole-Brain Activity Prediction in Zebrafish](https://mlsanity.com/paper/2503.02618/) (2025)
- [Decoding Layer Saliency in Language Transformers](https://mlsanity.com/paper/2308.05219/) (2023)
- [A Connectome Based Hexagonal Lattice Convolutional Network Model of the Drosophila Visual System](https://mlsanity.com/paper/1806.04793/) (2018)
- [Exploring Deep Learning Models for EEG Neural Decoding](https://mlsanity.com/paper/2503.16567/) (2025)

## Abstract (arXiv)

Delineating how animal behavior arises from neural activity is a foundational goal of neuroscience. However, as the computations underlying behavior unfold in networks of thousands of individual neurons across the entire brain, this presents challenges for investigating neural roles and computational mechanisms in large, densely wired mammalian brains during behavior. Transformers, the backbones of modern large language models (LLMs), have become powerful tools for neural decoding from smaller neural populations. These modern LLMs have benefited from extensive pre-training, and their sequence-to-sequence learning has been shown to generalize to novel tasks and data modalities, which may also confer advantages for neural decoding from larger, brain-wide activity recordings. Here, we present a systematic evaluation of off-the-shelf LLMs to decode behavior from brain-wide populations, termed NLP4Neuro, which we used to test LLMs on simultaneous calcium imaging and behavior recordings in larval zebrafish exposed to visual motion stimuli. Through NLP4Neuro, we found that LLMs become better at neural decoding when they use pre-trained weights learned from textual natural language data. Moreover, we found that a recent mixture-of-experts LLM, DeepSeek Coder-7b, significantly improved behavioral decoding accuracy, predicted tail movements over long timescales, and provided anatomically consistent highly interpretable readouts of neuron salience. NLP4Neuro demonstrates that LLMs are highly capable of informing brain-wide neural circuit dissection.