mlsanityData to 7 Oct 2026

FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inference

arXiv 2405.04065 ↗First posted 7 May 2024cs.CLcited by 1

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-Augmented Language Modeling (RALM) by integrating large language models (LLM) with relevant documents from an external corpus is a proven method for enabling the LLM to generate information beyond the scope of its pre-training corpus. Previous work utilizing retrieved content by simply prepending it to the input poses a high runtime issue, which degrades the inference efficiency of the LLMs because they fail to use the Key-Value (KV) cache efficiently. In this paper, we propose FlashBack, a modular RALM designed to improve the inference efficiency of RALM with appending context pattern while maintaining decent performance after fine-tuning by Low-Rank Adaption. FlashBack appends retrieved documents at the end of the context for efficiently utilizing the KV cache instead of prepending them. And we introduce Marking Token as two special prompt tokens for marking the boundary of the appending context during fine-tuning. Our experiments on testing generation quality show that FlashBack can remain decent generation quality in perplexity. And the inference speed of FlashBack is up to 4× faster than the prepending counterpart on a 7B LLM (Llama 2) in the runtime test. Via bypassing unnecessary re-computation, it demonstrates an advancement by achieving significantly faster inference speed, and this heightened efficiency will substantially reduce inferential cost.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 22 of its 31 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

22 of 31 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Reliable, Adaptable, and Attributable Language Models with Retrieval5 Mar
  2. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining11 Oct
  3. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuning2 Oct
  4. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models15 Aug
  5. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study13 Apr
  6. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  11. 2024Generative Representational Instruction Tuningcited by 227
  12. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
2 more
  1. 2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71
  2. 2023Context Compression for Auto-regressive Transformers with Sentinel Tokenscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026CogGuard: Cognitive and Operational Profiling for Proactive Warning in Edge Intelligent Servicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023In-Context Retrieval-Augmented Language Models
  2. 2023RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models
  3. 2024Inference Scaling for Bridging Retrieval and Augmented Generation
  4. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
  5. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculation
  6. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing
  7. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
  8. 2024Inference Scaling for Long-Context Retrieval Augmented Generation