# How Many Bytes Can You Take Out Of Brain-To-Text Decoding?

arXiv: https://arxiv.org/abs/2405.14055 · HTML: https://mlsanity.com/paper/2405.14055/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Linking artificial and human neural representations of language](https://mlsanity.com/paper/1910.01244/) (2019)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [EEG2TEXT: Open Vocabulary EEG-to-Text Decoding with EEG Pre-Training and Multi-View Transformer](https://mlsanity.com/paper/2405.02165/) (2024)
- [Brain2Word: Decoding Brain Activity for Language Generation](https://mlsanity.com/paper/2009.04765/) (2020)
- [Unveiling Thoughts: A Review of Advancements in EEG Brain Signal Decoding into Text](https://mlsanity.com/paper/2405.00726/) (2024)
- [Brain2Char: A Deep Architecture for Decoding Text from Brain Recordings](https://mlsanity.com/paper/1909.01401/) (2019)
- [Modeling correlated noise is necessary to decode uncertainty](https://mlsanity.com/paper/1708.04860/) (2017)

## Built on by (most cited first, 2 of 2)

- [A Common Measure of Communication for Speech Brain-Computer Interfaces](https://mlsanity.com/paper/2609.02887/) (2026)
- [Decoding Linguistic Representations of Human Brain](https://mlsanity.com/paper/2407.20622/) (2024)

## Abstract (arXiv)

Brain-computer interfaces have promising medical and scientific applications for aiding speech and studying the brain. In this work, we propose an information-based evaluation metric for brain-to-text decoders. Using this metric, we examine two methods to augment existing state-of-the-art continuous text decoders. We show that these methods, in concert, can improve brain decoding performance by upwards of 40% when compared to a baseline model. We further examine the informatic properties of brain-to-text decoders and show empirically that they have Zipfian power law dynamics. Finally, we provide an estimate for the idealized performance of an fMRI-based text decoder. We compare this idealized model to our current model, and use our information-based metric to quantify the main sources of decoding error. We conclude that a practical brain-to-text decoder is likely possible given further algorithmic improvements.