# Robust Implementation of Retrieval-Augmented Generation on Edge-based Computing-in-Memory Architectures

arXiv: https://arxiv.org/abs/2405.04700 · HTML: https://mlsanity.com/paper/2405.04700/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 7 May 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 20 of 20)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Positional Contrastive Learning for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2106.09157/) (2021)
- [News Category Dataset](https://mlsanity.com/paper/2209.11429/) (2022)
- [Device-Circuit-Architecture Co-Exploration for Computing-in-Memory Neural Accelerators](https://mlsanity.com/paper/1911.00139/) (2019)
- [PockEngine: Sparse and Efficient Fine-tuning in a Pocket](https://mlsanity.com/paper/2310.17752/) (2023)
- [Enabling On-Device Large Language Model Personalization with Self-Supervised Data Selection and Synthesis](https://mlsanity.com/paper/2311.12275/) (2023)
- [Uncertainty Modeling of Emerging Device-based Computing-in-Memory Neural Accelerators with Application to Neural Architecture Search](https://mlsanity.com/paper/2107.06871/) (2021)
- [SWIM: Selective Write-Verify for Computing-in-Memory Neural Accelerators](https://mlsanity.com/paper/2202.08395/) (2022)
- [When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs](https://mlsanity.com/paper/2309.14488/) (2023)
- [CorrectNet: Robustness Enhancement of Analog In-Memory Computing for Neural Networks by Error Suppression and Compensation](https://mlsanity.com/paper/2211.14917/) (2022)
- [FL-NAS: Towards Fairness of NAS for Resource Constrained Devices via Large Language Models](https://mlsanity.com/paper/2402.06696/) (2024)
- [Improving Realistic Worst-Case Performance of NVCiM DNN Accelerators through Training with Right-Censored Gaussian Noise](https://mlsanity.com/paper/2307.15853/) (2023)
- [Compute-in-Memory based Neural Network Accelerators for Safety-Critical Systems: Worst-Case Scenarios and Protections](https://mlsanity.com/paper/2312.06137/) (2023)
- [iMARS: An In-Memory-Computing Architecture for Recommendation Systems](https://mlsanity.com/paper/2202.09433/) (2022)

## Built on by (most cited first, 12 of 12)

- [Large Language Models have Intrinsic Self-Correction Ability](https://mlsanity.com/paper/2406.15673/) (2024)
- [Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices](https://mlsanity.com/paper/2406.03777/) (2024)
- [REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing](https://mlsanity.com/paper/2506.16444/) (2025)
- [Tiny-Align: Bridging Automatic Speech Recognition and Large Language Model on the Edge](https://mlsanity.com/paper/2411.13766/) (2024)
- [TPU-Gen: LLM-Driven Custom Tensor Processing Unit Generator](https://mlsanity.com/paper/2503.05951/) (2025)
- [NVCiM-PT: An NVCiM-assisted Prompt Tuning Framework for Edge LLMs](https://mlsanity.com/paper/2411.08244/) (2024)
- [DIRC-RAG: Accelerating Edge RAG with Robust High-Density and High-Loading-Bandwidth Digital In-ReRAM Computation](https://mlsanity.com/paper/2510.25278/) (2025)
- [CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation](https://mlsanity.com/paper/2510.19670/) (2025)
- [CiMRAG: CiM-Aware Domain-Adaptive and Noise-Resilient Retrieval-Augmented Generation for Edge-Based LLMs](https://mlsanity.com/paper/2601.20041/) (2026)
- [CQ-CiM: Hardware-Aware Embedding Shaping for Robust CiM-Based Retrieval](https://mlsanity.com/paper/2602.20083/) (2026)
- [From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG](https://mlsanity.com/paper/2608.19535/) (2026)
- [MOMAT: Mixture of Multiple Atlases for Low-Power Jailbreak Defense of Quantized LLMs](https://mlsanity.com/paper/2610.01058/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) deployed on edge devices learn through fine-tuning and updating a certain portion of their parameters. Although such learning methods can be optimized to reduce resource utilization, the overall required resources remain a heavy burden on edge devices. Instead, Retrieval-Augmented Generation (RAG), a resource-efficient LLM learning method, can improve the quality of the LLM-generated content without updating model parameters. However, the RAG-based LLM may involve repetitive searches on the profile data in every user-LLM interaction. This search can lead to significant latency along with the accumulation of user data. Conventional efforts to decrease latency result in restricting the size of saved user data, thus reducing the scalability of RAG as user data continuously grows. It remains an open question: how to free RAG from the constraints of latency and scalability on edge devices? In this paper, we propose a novel framework to accelerate RAG via Computing-in-Memory (CiM) architectures. It accelerates matrix multiplications by performing in-situ computation inside the memory while avoiding the expensive data transfer between the computing unit and memory. Our framework, Robust CiM-backed RAG (RoCR), utilizing a novel contrastive learning-based training method and noise-aware training, can enable RAG to efficiently search profile data with CiM. To the best of our knowledge, this is the first work utilizing CiM to accelerate RAG.