mlsanityData to 7 Oct 2026

Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts

arXiv 2404.02022 ↗First posted 2 Apr 2024cs.CLcited by 3

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

In the era of large language models, applying techniques such as Retrieval Augmented Generation can better address Open-Domain Question-Answering problems. Due to constraints including model sizes and computing resources, the length of context is often limited, and it becomes challenging to empower the model to cover overlong contexts while answering questions from open domains. This paper proposes a general and convenient method to covering longer contexts in Open-Domain Question-Answering tasks. It leverages a small encoder language model that effectively encodes contexts, and the encoding applies cross-attention with origin inputs. With our method, the origin language models can cover several times longer contexts while keeping the computing requirements close to the baseline. Our experiments demonstrate that after fine-tuning, there is improved performance across two held-in datasets, four held-out datasets, and also in two In Context Learning settings.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 28 of its 46 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

28 of 46 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
16 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  2. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  5. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  6. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2019ELI5: Long Form Question Answeringcited by 567
  9. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  10. 2023Adapting Language Models to Compress Contextscited by 350
  11. 2024Long Context Compression with Activation Beaconcited by 101
  12. 2021Improving Named Entity Recognition by External Context Retrieving and Cooperative Learningcited by 76
  13. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generatorcited by 68
  14. 2018Towards Exploiting Background Knowledge for Building Conversation Systemscited by 57
  15. 2022Visconde: Multi-document QA with GPT-3 and Neural Rerankingcited by 23
  16. 2022Few-shot Reranking for Multi-hop QA via Language Model Promptingcited by 19

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Bias-Aware Agent: Enhancing Fairness in AI-Driven Knowledge Retrievalcited by 5
  2. 2025OnRL-RAG: Real-Time Personalized Mental Health Dialogue Systemcited by 1
  3. 2025HANRAG: Heuristic Accurate Noise-resistant Retrieval-Augmented Generation for Multi-hop Question Answeringcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering
  2. 2024Context Awareness Gate For Retrieval Augmented Generation
  3. 2024RE-RAG: Improving Open-Domain QA Performance and Interpretability with Relevance Estimator in Retrieval-Augmented Generation
  4. 2025Provence: efficient and robust context pruning for retrieval-augmented generation
  5. 2022Decoupled Context Processing for Context Augmented Language Modeling
  6. 2024Retrieval Augmented Generation for Domain-specific Question Answering
  7. 2024Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning
  8. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context