mlsanityData to 14 Oct 2026

ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion

arXiv 2401.02993 ↗First posted 4 Jan 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Retrieval-based augmentations (RA) incorporating knowledge from an external database into language models have greatly succeeded in various knowledge-intensive (KI) tasks. However, integrating retrievals in non-knowledge-intensive (NKI) tasks is still challenging. Existing works focus on concatenating retrievals with inputs to improve model performance. Unfortunately, the use of retrieval concatenation-based augmentations causes an increase in the input length, substantially raising the computational demands of attention mechanisms. This paper proposes a new paradigm of RA named ReFusion, a computation-efficient Retrieval representation Fusion with bi-level optimization. Unlike previous works, ReFusion directly fuses the retrieval representations into the hidden states of models. Specifically, ReFusion leverages an adaptive retrieval integrator to seek the optimal combination of the proposed ranking schemes across different model layers. Experimental results demonstrate that the proposed ReFusion can achieve superior and robust performance in various NKI tasks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 4 Jan 2024.

Builds on

33 of 42 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  9. 2017Billion-scale similarity search with GPUscited by 2,497
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
21 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  2. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  4. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  5. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  6. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 298
  7. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  8. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  9. 2022DocPrompting: Generating Code by Retrieving the Docscited by 153
  10. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  11. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
  12. 2014Learning Ordered Representations with Nested Dropoutcited by 84
  13. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  14. 2019Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answeringcited by 48
  15. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learningcited by 36
  16. 2022RACE: Retrieval-Augmented Commit Message Generationcited by 27
  17. 2021FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text modelscited by 26
  18. 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
  19. 2023Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Taskscited by 21
  20. 2022Retrieval-Augmented Generative Question Answering for Event Argument Extractioncited by 18
  21. 2021Variational Nested Dropoutcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 2.

5 papers.

  1. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 71
  2. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inferencecited by 4
  3. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Managementcited by 1
  4. 2025Beyond Semantic Similarity: Reducing Unnecessary API Calls via Behavior-Aligned Retrievercited by 0
  5. 2026ReFilter: Improving Robustness of Retrieval-Augmented Generation via Gated Filtercited by 0