Knowledge Base Index Compression via Dimensionality and Precision Reduction
Not on the map. Why?
Abstract (arXiv)
Recently neural network based approaches to knowledge-intensive NLP tasks, such as question answering, started to rely heavily on the combination of neural retrievers and readers. Retrieval is typically performed over a large textual knowledge base (KB) which requires significant memory and compute resources, especially when scaled up. On HotpotQA we systematically investigate reducing the size of the KB index by means of dimensionality (sparse random projections, PCA, autoencoders) and numerical precision reduction. Our results show that PCA is an easy solution that requires very little data and is only slightly worse than autoencoders, which are less stable. All methods are sensitive to pre- and post-processing and data should always be centered and normalized both before and after dimension reduction. Finally, we show that it is possible to combine PCA with using 1bit per dimension. Overall we achieve (1) 100× compression with 75%, and (2) 24× compression with 92% original retrieval performance.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 6 Apr 2022.
Builds on
22 of 40 references.
References · 22
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2017Billion-scale similarity search with GPUscited by 2,497
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
- 2021Improving language models by retrieving from trillions of tokenscited by 1,293
- 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
- 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
- 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
- 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
10 more
- 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
- 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 156
- 2021Efficient Nearest Neighbor Language Modelscited by 93
- 2021Adaptive Semiparametric Language Modelscited by 83
- 2021Efficient Passage Retrieval with Hashing for Open-domain Question Answeringcited by 53
- 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
- 2020A Memory Efficient Baseline for Open Domain Question Answeringcited by 32
- 2021A Proposed Conceptual Framework for a Representational Approach to Information Retrievalcited by 24
- 2013Barnes-Hut-SNEcited by 14
- 2022Artefact Retrieval: Overview of NLP Models with Knowledge Base Accesscited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1.
2 papers.