# Brain-to-Text Decoding with Context-Aware Neural Representations and Large Language Models

arXiv: https://arxiv.org/abs/2411.10657 · HTML: https://mlsanity.com/paper/2411.10657/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2024.

## Built on by (most cited first, 7 of 7)

- [Brain-to-Text Benchmark '24: Lessons Learned](https://mlsanity.com/paper/2412.17227/) (2024)
- [SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding](https://mlsanity.com/paper/2507.08402/) (2025)
- [A cross-species neural foundation model for end-to-end speech decoding](https://mlsanity.com/paper/2511.21740/) (2025)
- [Decoding the decoder: Contextual sequence-to-sequence modeling for intracortical speech decoding](https://mlsanity.com/paper/2603.20246/) (2026)
- [MoDAl: Self-Supervised Neural Modality Discovery via Decorrelation for Speech Neuroprosthesis](https://mlsanity.com/paper/2605.00025/) (2026)
- [Phoneme- vs. Character-Level Targets and Selective State-Space Models for Intracortical Brain-to-Text](https://mlsanity.com/paper/2607.26751/) (2026)
- [Myovox: Reading Speech from the Muscles of the Face](https://mlsanity.com/paper/2609.17548/) (2026)

## Abstract (arXiv)

Decoding attempted speech from neural activity offers a promising avenue for restoring communication abilities in individuals with speech impairments. Previous studies have focused on mapping neural activity to text using phonemes as the intermediate target. While successful, decoding neural activity directly to phonemes ignores the context dependent nature of the neural activity-to-phoneme mapping in the brain, leading to suboptimal decoding performance. In this work, we propose the use of diphone - an acoustic representation that captures the transitions between two phonemes - as the context-aware modeling target. We integrate diphones into existing phoneme decoding frameworks through a novel divide-and-conquer strategy in which we model the phoneme distribution by marginalizing over the diphone distribution. Our approach effectively leverages the enhanced context-aware representation of diphones while preserving the manageable class size of phonemes, a key factor in simplifying the subsequent phoneme-to-text conversion task. We demonstrate the effectiveness of our approach on the Brain-to-Text 2024 benchmark, where it achieves state-of-the-art Phoneme Error Rate (PER) of 15.34% compared to 16.62% PER of monophone-based decoding. When coupled with finetuned Large Language Models (LLMs), our method yields a Word Error Rate (WER) of 5.77%, significantly outperforming the 8.93% WER of the leading method in the benchmark.