mlsanityData to 7 Oct 2026

BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models

arXiv 2310.01329 ↗First posted 2 Oct 2023cs.CL cs.AIcited by 5

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval augmentation addresses many critical problems in large language models such as hallucination, staleness, and privacy leaks. However, running retrieval-augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages, significantly reducing computation during inference. Despite the potential loss of accuracy, our new calibration techniques and training objectives restore performance. Combined with offline and runtime compression, this only requires 127GB of disk space for encoding 3 billion tokens in Wikipedia. Our experiments show that on five knowledge-intensive NLP tasks, BTR accelerates state-of-the-art inference by up to 4x and reduces storage by over 100x while maintaining over 95% task performance.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 41 of its 50 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

41 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  2. 2023Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute25 Jan
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories20 Dec
  4. 2022FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference15 Dec
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
22 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  4. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  5. 2022Token Merging: Your ViT But Fastercited by 995
  6. 2020On Layer Normalization in the Transformer Architecturecited by 906
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  9. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  10. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  11. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  12. 2017HashNet: Deep Learning to Hash by Continuationcited by 121
  13. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  14. 2020Learning Dense Representations of Phrases at Scalecited by 91
  15. 2022BiBERT: Accurate Fully Binarized BERTcited by 70
  16. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  17. 2021Efficient Passage Retrieval with Hashing for Open-domain Question Answeringcited by 53
  18. 2022Nonparametric Masked Language Modelingcited by 39
  19. 2020DeFormer: Decomposing Pre-trained Transformers for Faster Question Answeringcited by 29
  20. 2018Near-lossless Binarization of Word Embeddingscited by 13
  21. 2017SUBIC: A supervised, structured binary code for image searchcited by 12
  22. 2019Learning Compressed Sentence Representations for On-Device Text Processingcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 1.

5 papers.

  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  2. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  3. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 54
  4. 20241.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Datacited by 6
  5. 2025Mathematical Analysis of Hallucination Dynamics in Large Language Models: Uncertainty Quantification, Advanced Decoding, and Principled Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Latent Abstraction for Retrieval-Augmented Generation
  2. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models
  3. 2024One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models
  4. 2025ARC-Encoder: learning compressed text representations for large language models
  5. 2025Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation
  6. 2025Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity
  7. 2025Long-Tail Crisis in Nearest Neighbor Language Models
  8. 2023TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction