mlsanityData to 7 Oct 2026

Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models

arXiv 2310.09949 ↗First posted 15 Oct 2023cs.LG cs.AI cs.ARcited by 23

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

A Retrieval-Augmented Language Model (RALM) combines a large language model (LLM) with a vector database to retrieve context-specific knowledge during text generation. This strategy facilitates impressive generation quality even with smaller models, thus reducing computational demands by orders of magnitude. To serve RALMs efficiently and flexibly, we propose Chameleon, a heterogeneous accelerator system integrating both LLM and vector search accelerators in a disaggregated architecture. The heterogeneity ensures efficient serving for both inference and retrieval, while the disaggregation allows independent scaling of LLM and vector search accelerators to fulfill diverse RALM requirements. Our Chameleon prototype implements vector search accelerators on FPGAs and assigns LLM inference to GPUs, with CPUs as cluster coordinators. Evaluated on various RALMs, Chameleon exhibits up to 2.16× reduction in latency and 3.18x speedup in throughput compared to the hybrid CPU-GPU architecture. The promising results pave the way for adopting heterogeneous accelerators for not only LLM inference but also vector search in future RALM systems.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 60 of its 158 references that are in the map. First posted on arXiv 15 Oct 2023.

Builds on

60 of 158 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
38 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2019Machine Unlearningcited by 1,020
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  10. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  13. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  14. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  15. 2021Internet-Augmented Dialogue Generationcited by 217
  16. 2020Nearest Neighbor Machine Translationcited by 209
  17. 2020When Machine Unlearning Jeopardizes Privacycited by 181
  18. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  19. 2017Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graphcited by 103
  20. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  21. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 97
  22. 2020Pre-training via Paraphrasingcited by 92
  23. 2021Adaptive Semiparametric Language Modelscited by 83
  24. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answeringcited by 75
  25. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  26. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  27. 2021Fast Nearest Neighbor Machine Translationcited by 37
  28. 2021RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inferencecited by 35
  29. 2021SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Searchcited by 35
  30. 2023Why do Nearest Neighbor Language Models Work?cited by 29
  31. 2023High-Dimensional Approximate Nearest Neighbor Search: with Reliable and Efficient Distance Comparison Operationscited by 25
  32. 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
  33. 2023The Dark Side of ChatGPT: Legal and Ethical Challenges from Stochastic Parrots and Hallucinationcited by 23
  34. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
  35. 2023High-Throughput Vector Similarity Search in Knowledge Graphscited by 19
  36. 2022TPU-KNN: K Nearest Neighbor Search at Peak FLOP/scited by 18
  37. 2022Efficient Quantized Sparse Matrix Operations on Tensor Corescited by 16
  38. 2022CFU Playground: Full-Stack Open-Source Framework for Tiny Machine Learning (tinyML) Acceleration on FPGAscited by 11

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 5.

23 papers.

Sort:

By area:

  1. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  2. 2025In-depth Analysis of Graph-based RAG in a Unified Frameworkcited by 45
  3. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-designcited by 28
  4. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 18
  5. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  6. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  7. 2024CANDY: A Benchmark for Continuous Approximate Nearest Neighbor Search with Dynamic Data Ingestioncited by 4
  8. 2025Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Searchcited by 4
  9. 2025MIRAGE: Misleading Retrieval-Augmented Generation via Black-box and Query-agnostic Poisoning Attackscited by 4
  10. 2026Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memorycited by 4
  11. 2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAGcited by 3
  12. 2025HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Servingcited by 3
11 more
  1. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPUcited by 2
  2. 2025Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Searchcited by 2
  3. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 2
  4. 2025LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialoguescited by 1
  5. 2025PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devicescited by 1
  6. 2024UpANNS: Enhancing Billion-Scale ANNS Efficiency with Real-World PIM Architecturecited by 0
  7. 2025Harmonia: End-to-End RAG Serving Optimizationcited by 0
  8. 2025All-in-one Graph-based Indexing for Hybrid Search on GPUscited by 0
  9. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  10. 2026RAG-Stack: Co-Optimizing RAG Serving Performance and Qualitycited by 0
  11. 2026RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  2. 2024Accelerating Retrieval-Augmented Generation
  3. 2024JORA: JAX Tensor-Parallel LoRA Library for Retrieval Augmented Fine-Tuning
  4. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  5. 2025AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM
  6. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  7. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment
  8. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing