# Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding

arXiv: https://arxiv.org/abs/2507.02800 · HTML: https://mlsanity.com/paper/2507.02800/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2025.

## Built on by (most cited first, 6 of 6)

- [A cross-species neural foundation model for end-to-end speech decoding](https://mlsanity.com/paper/2511.21740/) (2025)
- [Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings](https://mlsanity.com/paper/2608.18114/) (2026)
- [Brain-to-Language Decoding: Tasks, Signals, Methods, Evaluation, Practical Use and Beyond](https://mlsanity.com/paper/2609.27650/) (2026)
- [ALIGN: Adversarial Learning for Generalizable Speech Neuroprosthesis](https://mlsanity.com/paper/2603.18299/) (2026)
- [From Neurons to Conversation: Speech Brain-Computer Interfaces](https://mlsanity.com/paper/2609.36736/) (2026)
- [NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings](https://mlsanity.com/paper/2610.02864/) (2026)

## Abstract (arXiv)

Speech neuroprostheses aim to restore communication for people with severe paralysis by decoding speech directly from neural activity. To accelerate algorithmic progress, a recent benchmark released intracranial recordings from a paralyzed participant attempting to speak, along with a baseline decoding algorithm. Prior work on the benchmark showed impressive accuracy gains. However, these gains increased computational costs and were not demonstrated in a real-time decoding setting. Here, we make three contributions that pave the way towards accurate, efficient, and real-time neural speech decoding. First, we incorporate large amounts of time-masking during training. On average, over 50% of each trial is masked. Second, we replace the gated recurrent unit (GRU) architecture used in the baseline algorithm with a compact Transformer. The Transformer architecture uses 83% fewer parameters, cuts peak GPU memory usage by 52%, and is significantly faster to calibrate relative to the GRU. Third, we design a lightweight variant of an existing test-time adaptation method developed for decoding handwriting from neural activity. Our variant adapts the model using multiple time-masked augmentations of a single trial and requires only one gradient step per trial. Together, these contributions reduce word error rate by over 20% and effectively mitigate performance degradations across held-out days in a real-time decoding setting while substantially lowering computational costs.