mlsanityData to 7 Oct 2026

Retro-li: Small-Scale Retrieval Augmented Generation Supporting Noisy Similarity Searches and Domain Shift Generalization

arXiv 2410.00004 ↗First posted 12 Sep 2024cs.IR cs.AI cs.CLcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

The retrieval augmented generation (RAG) system such as Retro has been shown to improve language modeling capabilities and reduce toxicity and hallucinations by retrieving from a database of non-parametric memory containing trillions of entries. We introduce Retro-li that shows retrieval can also help using a small-scale database, but it demands more accurate and better neighbors when searching in a smaller hence sparser non-parametric memory. This can be met by using a proper semantic similarity search. We further propose adding a regularization to the non-parametric memory for the first time: it significantly reduces perplexity when the neighbor search operations are noisy during inference, and it improves generalization when a domain shift occurs. We also show that Retro-li's non-parametric memory can potentially be implemented on analog in-memory computing hardware, exhibiting O(1) search time while causing noise in retrieving neighbors, with minimal (<1%) performance loss. Our code is available at: https://github.com/IBM/Retrieval-Enhanced-Transformer-Little.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 38 of its 56 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

38 of 56 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
26 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2015Teaching Machines to Read and Comprehendcited by 1,039
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  10. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  11. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  13. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  14. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  15. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  16. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  17. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  18. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  19. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  20. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  21. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  22. 2023Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based acceleratorscited by 42
  23. 2021A flexible and fast PyTorch toolkit for simulating training and inference on analog crossbar arrayscited by 38
  24. 2020Robust High-dimensional Memory-augmented Neural Networkscited by 31
  25. 2023Meta-training with Demonstration Retrieval for Efficient Few-shot Learningcited by 2
  26. 2023Retrieval-Augmented Meta Learning for Low-Resource Text Classificationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Generalization Ability of Retrieval-Enhanced Transformers
  2. 2024Accelerating Retrieval-Augmented Generation
  3. 2024GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
  4. 2026D-NOVA: In-Storage Retrieval Accelerator via Dual-Bound 3D NAND-Optimized Similarity Search with Vector Adaptation
  5. 2026A Dynamic Retrieval-Augmented Generation System with Selective Memory and Remembrance
  6. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  7. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing
  8. 2026Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory