# ReadOnce Transformers: Reusable Representations of Text for Transformers

arXiv: https://arxiv.org/abs/2010.12854 · HTML: https://mlsanity.com/paper/2010.12854/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 4 of 4)

- [Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs](https://mlsanity.com/paper/2211.07950/) (2022)
- [Embedding Recycling for Language Models](https://mlsanity.com/paper/2207.04993/) (2022)
- [CoLLEGe: Concept Embedding Generation for Large Language Models](https://mlsanity.com/paper/2403.15362/) (2024)
- [CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals](https://mlsanity.com/paper/2607.28263/) (2026)

## Abstract (arXiv)

We present ReadOnce Transformers, an approach to convert a transformer-based model into one that can build an information-capturing, task-independent, and compressed representation of text. The resulting representation is reusable across different examples and tasks, thereby requiring a document shared across many examples or tasks to only be read once. This leads to faster training and evaluation of models. Additionally, we extend standard text-to-text transformer models to Representation+Text-to-text models, and evaluate on multiple downstream tasks: multi-hop QA, abstractive QA, and long-document summarization. Our one-time computed representation results in a 2x-5x speedup compared to standard text-to-text models, while the compression also allows existing language models to handle longer documents without the need for designing new pre-trained models.