# Brain-to-Text Benchmark '24: Lessons Learned

arXiv: https://arxiv.org/abs/2412.17227 · HTML: https://mlsanity.com/paper/2412.17227/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling](https://mlsanity.com/paper/2403.03234/) (2024)
- [On Empirical Comparisons of Optimizers for Deep Learning](https://mlsanity.com/paper/1910.05446/) (2019)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Making brain-machine interfaces robust to future neural variability](https://mlsanity.com/paper/1610.05872/) (2016)
- [Enabling hyperparameter optimization in sequential autoencoders for spiking neural data](https://mlsanity.com/paper/1908.07896/) (2019)
- [A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition](https://mlsanity.com/paper/2403.05583/) (2024)
- [Brain-to-Text Decoding with Context-Aware Neural Representations and Large Language Models](https://mlsanity.com/paper/2411.10657/) (2024)
- [Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models](https://mlsanity.com/paper/2406.11568/) (2024)

## Built on by (most cited first, 8 of 8)

- [The 2025 PNPL Competition: Speech Detection and Phoneme Classification in the LibriBrain Dataset](https://mlsanity.com/paper/2506.10165/) (2025)
- [Generalizable, real-time neural decoding with hybrid state-space models](https://mlsanity.com/paper/2506.05320/) (2025)
- [A cross-species neural foundation model for end-to-end speech decoding](https://mlsanity.com/paper/2511.21740/) (2025)
- [A Common Measure of Communication for Speech Brain-Computer Interfaces](https://mlsanity.com/paper/2609.02887/) (2026)
- [The 2026 PNPL Competition: Word Classification and Efficient Cross-Subject Generalisation in LibriBrain100](https://mlsanity.com/paper/2609.03231/) (2026)
- [The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding](https://mlsanity.com/paper/2609.10296/) (2026)
- [Brain-to-Language Decoding: Tasks, Signals, Methods, Evaluation, Practical Use and Beyond](https://mlsanity.com/paper/2609.27650/) (2026)
- [Phoneme- vs. Character-Level Targets and Selective State-Space Models for Intracortical Brain-to-Text](https://mlsanity.com/paper/2607.26751/) (2026)

## Abstract (arXiv)

Speech brain-computer interfaces aim to decipher what a person is trying to say from neural activity alone, restoring communication to people with paralysis who have lost the ability to speak intelligibly. The Brain-to-Text Benchmark '24 and associated competition was created to foster the advancement of decoding algorithms that convert neural activity to text. Here, we summarize the lessons learned from the competition ending on June 1, 2024 (the top 4 entrants also presented their experiences in a recorded webinar). The largest improvements in accuracy were achieved using an ensembling approach, where the output of multiple independent decoders was merged using a fine-tuned large language model (an approach used by all 3 top entrants). Performance gains were also found by improving how the baseline recurrent neural network (RNN) model was trained, including by optimizing learning rate scheduling and by using a diphone training objective. Improving upon the model architecture itself proved more difficult, however, with attempts to use deep state space models or transformers not yet appearing to offer a benefit over the RNN baseline. The benchmark will remain open indefinitely to support further work towards increasing the accuracy of brain-to-text algorithms.