mlsanityData to 7 Oct 2026

4bit-Quantization in Vector-Embedding for RAG

arXiv 2501.10534 ↗First posted 17 Jan 2025cs.LG cs.AIcited by 4

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-augmented generation (RAG) is a promising technique that has shown great potential in addressing some of the limitations of large language models (LLMs). LLMs have two major limitations: they can contain outdated information due to their training data, and they can generate factually inaccurate responses, a phenomenon known as hallucinations. RAG aims to mitigate these issues by leveraging a database of relevant documents, which are stored as embedding vectors in a high-dimensional space. However, one of the challenges of using high-dimensional embeddings is that they require a significant amount of memory to store. This can be a major issue, especially when dealing with large databases of documents. To alleviate this problem, we propose the use of 4-bit quantization to store the embedding vectors. This involves reducing the precision of the vectors from 32-bit floating-point numbers to 4-bit integers, which can significantly reduce the memory requirements. Our approach has several benefits. Firstly, it significantly reduces the memory storage requirements of the high-dimensional vector database, making it more feasible to deploy RAG systems in resource-constrained environments. Secondly, it speeds up the searching process, as the reduced precision of the vectors allows for faster computation. Our code is available at https://github.com/taeheej/4bit-Quantization-in-Vector-Embedding-for-RAG

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 20 of its 27 references that are in the map. First posted on arXiv 17 Jan 2025.

Builds on

20 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  9. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  11. 2024Large Language Models: A Surveycited by 506
  12. 2021A White Paper on Neural Network Quantizationcited by 453
7 more
  1. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  2. 2023GraphGPT: Graph Instruction Tuning for Large Language Modelscited by 223
  3. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  4. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  5. 2018Approximate Nearest Neighbor Search in High Dimensionscited by 45
  6. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  7. 2021What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Studycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Cache Mechanism for Agent RAG Systemscited by 5
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inferencecited by 3
  3. 2026When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compressioncited by 1
  4. 2026Reinforcement-Learned Unequal Error Protection for Quantized Semantic Embeddingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimization of embeddings storage for RAG systems using quantization and dimensionality reduction techniques
  2. 2026Progressive Searching for Retrieval in RAG
  3. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  4. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
  5. 2024R^3AG: First Workshop on Refined and Reliable Retrieval Augmented Generation
  6. 2026Matryoshka Hash Representations for Model-Aware Compact Semantic Retrieval
  7. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  8. 2024One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models