mlsanityData to 7 Oct 2026

Suffix Retrieval-Augmented Language Modeling

arXiv 2211.03053 ↗First posted 6 Nov 2022cs.CLcited by 0

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Causal language modeling (LM) uses word history to predict the next word. BERT, on the other hand, makes use of bi-directional word information in a sentence to predict words at masked positions. While BERT is effective in sequence encoding, it is non-causal by nature and is not designed for sequence generation. In this paper, we propose a novel language model, SUffix REtrieval-Augmented LM (SUREALM), that simulates a bi-directional contextual effect in an autoregressive manner. SUREALM employs an embedding retriever to search for training sentences in a data store that share similar word history during sequence generation. In particular, the suffix portions of the retrieved sentences mimick the "future" context. We evaluated our proposed model on the DSTC9 spoken dialogue corpus and showed promising word perplexity reduction on the validation and test set compared to competitive baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 12 of its 17 references that are in the map. First posted on arXiv 6 Nov 2022.

Builds on

12 of 17 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 10

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  7. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  8. 2014Memory Networkscited by 354
  9. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  10. 2021Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Access Track in DSTC9cited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Assessing "Implicit" Retrieval Robustness of Large Language Models
  2. 2021Improving language models by retrieving from trillions of tokens
  3. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey
  4. 2026Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
  5. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context
  6. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
  7. 2024Retrieval Augmented End-to-End Spoken Dialog Models
  8. 2024More Room for Language: Investigating the Effect of Retrieval on Language Models