MEMORY-VQ: Compression for Tractable Internet-Scale Memory
Not on the map. Why?
Abstract (arXiv)
Retrieval augmentation is a powerful but expensive method to make language models more knowledgeable about the world. Memory-based methods like LUMEN pre-compute token representations for retrieved passages to drastically speed up inference. However, memory also leads to much greater storage requirements from storing pre-computed representations. We propose MEMORY-VQ, a new method to reduce storage requirements of memory-augmented models without sacrificing performance. Our method uses a vector quantization variational autoencoder (VQ-VAE) to compress token representations. We apply MEMORY-VQ to the LUMEN model to obtain LUMEN-VQ, a memory model that achieves a 16x compression rate with comparable performance on the KILT benchmark. LUMEN-VQ enables practical retrieval augmentation even for extremely large retrieval corpora.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 28 Aug 2023.
Builds on
34 of 46 references.
References · 34
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2017Neural Discrete Representation Learningcited by 4,650
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
- 2017Billion-scale similarity search with GPUscited by 2,477
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
- 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
- 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
- 2021Improving language models by retrieving from trillions of tokenscited by 1,278
- 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
- 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
22 more
- 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
- 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
- 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
- 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
- 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
- 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
- 2022Memorizing Transformerscited by 219
- 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
- 2022Training Language Models with Memory Augmentationcited by 112
- 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
- 2020Hierarchical Quantized Autoencoderscited by 55
- 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
- 2022An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Taskscited by 36
- 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
- 2023Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your computecited by 17
- 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14
- 2022Multi-Task Retrieval-Augmented Text Generation with Relevance Samplingcited by 10
- 2021SDR: Efficient Neural Re-ranking using Succinct Document Representationcited by 7
- 2023LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interactioncited by 7
- 2023GLIMMER: generalized late-interaction memory rerankercited by 7
- 2021Composite Re-Ranking for Efficient Document Search with BERTcited by 5
- 2022Compact Token Representations with Contextual Quantization for Efficient Document Re-rankingcited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.