# Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs

arXiv: https://arxiv.org/abs/2502.12462 · HTML: https://mlsanity.com/paper/2502.12462/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 16 of its 17 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 16 of 16)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://mlsanity.com/paper/2201.03533/) (2022)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [One Thousand and One Pairs: A "novel" challenge for long-context language models](https://mlsanity.com/paper/2406.16264/) (2024)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [DocFinQA: A Long-Context Financial Reasoning Dataset](https://mlsanity.com/paper/2401.06915/) (2024)
- [Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?](https://mlsanity.com/paper/2406.13121/) (2024)
- [Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models](https://mlsanity.com/paper/2406.11230/) (2024)
- [LongHealth: A Question Answering Benchmark with Long Clinical Documents](https://mlsanity.com/paper/2401.14490/) (2024)
- [NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens](https://mlsanity.com/paper/2403.12766/) (2024)
- [CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models](https://mlsanity.com/paper/2403.03514/) (2024)

## Built on by (most cited first, 9 of 9)

- [Gated Differentiable Working Memory for Long-Context Language Modeling](https://mlsanity.com/paper/2601.12906/) (2026)
- [Reflexive Prompt Engineering: A Framework for Responsible Prompt Engineering and Interaction Design](https://mlsanity.com/paper/2504.16204/) (2025)
- [An Agile Method for Implementing Retrieval Augmented Generation Tools in Industrial SMEs](https://mlsanity.com/paper/2508.21024/) (2025)
- [Retrieval-Augmented Generation for Electrocardiogram-Language Models](https://mlsanity.com/paper/2510.00261/) (2025)
- [MPCAR: Multi-Perspective Contextual Augmentation for Enhanced Visual Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2508.12400/) (2025)
- [A Study on the Impact of Fault localization Granularity for Repository-Scale Code Repair Tasks](https://mlsanity.com/paper/2604.00167/) (2026)
- [RAG-Enhanced Large Language Models for Dynamic Content Expiration Prediction in Web Search](https://mlsanity.com/paper/2605.13052/) (2026)
- [PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding](https://mlsanity.com/paper/2506.17310/) (2025)
- [FLARE: Few-shot Learning-based Adaptive Reflective Engine](https://mlsanity.com/paper/2608.02919/) (2026)

## Abstract (arXiv)

This paper addresses the challenge of comprehending very long contexts in Large Language Models (LLMs) by proposing a method that emulates Retrieval Augmented Generation (RAG) through specialized prompt engineering and chain-of-thought (CoT) reasoning. While recent LLMs support over 100,000 tokens in a single prompt, simply enlarging context windows has not guaranteed robust multi-hop reasoning when key details are scattered across massive input. Our approach treats the model as both the retriever and the reasoner: it first tags relevant segments within a long passage, then employs a stepwise CoT workflow to integrate these pieces of evidence. This single-pass method thereby reduces reliance on an external retriever, yet maintains focus on crucial segments. We evaluate our approach on selected tasks from BABILong, which interleaves standard bAbI QA problems with large amounts of distractor text. Compared to baseline (no retrieval) and naive RAG pipelines, our approach more accurately handles multi-fact questions such as object location tracking, counting, and indefinite knowledge. Furthermore, we analyze how prompt structure, including the order of question, relevant-text tags, and overall instructions, significantly affects performance. These findings underscore that optimized prompt engineering, combined with guided reasoning, can enhance LLMs' long-context comprehension and serve as a lightweight alternative to traditional retrieval pipelines.