# LLM In-Context Recall is Prompt Dependent

arXiv: https://arxiv.org/abs/2404.08865 · HTML: https://mlsanity.com/paper/2404.08865/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 9 references that are in the map.
- First posted on arXiv 13 Apr 2024.

## Builds on (most cited first, 2 of 2)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)

## Built on by (most cited first, 15 of 15)

- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly](https://mlsanity.com/paper/2408.01866/) (2024)
- [Language Models' Factuality Depends on the Language of Inquiry](https://mlsanity.com/paper/2502.17955/) (2025)
- [SynthTRIPs: A Knowledge-Grounded Framework for Benchmark Query Generation for Personalized Tourism Recommenders](https://mlsanity.com/paper/2504.09277/) (2025)
- [Statistical Advantage of Softmax Attention: Insights from Single-Location Regression](https://mlsanity.com/paper/2509.21936/) (2025)
- [Attention Instruction: Amplifying Attention in the Middle via Prompting](https://mlsanity.com/paper/2406.17095/) (2024)
- [Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models](https://mlsanity.com/paper/2409.04318/) (2024)
- [Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"](https://mlsanity.com/paper/2502.06898/) (2025)
- [RAGSys: Item-Cold-Start Recommender as RAG System](https://mlsanity.com/paper/2405.17587/) (2024)
- [Teaching LLMs Music Theory with In-Context Learning and Chain-of-Thought Prompting: Pedagogical Strategies for Machines](https://mlsanity.com/paper/2503.22853/) (2025)
- [UrzaGPT: LoRA-Tuned Large Language Models for Card Selection in Collectible Card Games](https://mlsanity.com/paper/2508.08382/) (2025)
- [A Survey on Training-free Alignment of Large Language Models](https://mlsanity.com/paper/2508.09016/) (2025)
- [Testing Uncertainty of Large Language Models for Physics Knowledge and Reasoning](https://mlsanity.com/paper/2411.14465/) (2024)
- [Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks](https://mlsanity.com/paper/2412.00546/) (2024)
- [Contractual Deepfakes: Can Large Language Models Generate Contracts?](https://mlsanity.com/paper/2602.09384/) (2026)

## Abstract (arXiv)

The proliferation of Large Language Models (LLMs) highlights the critical importance of conducting thorough evaluations to discern their comparative advantages, limitations, and optimal use cases. Particularly important is assessing their capacity to accurately retrieve information included in a given prompt. A model's ability to do this significantly influences how effectively it can utilize contextual details, thus impacting its practical efficacy and dependability in real-world applications. Our research analyzes the in-context recall performance of various LLMs using the needle-in-a-haystack method. In this approach, a factoid (the "needle") is embedded within a block of filler text (the "haystack"), which the model is asked to retrieve. We assess the recall performance of each model across various haystack lengths and with varying needle placements to identify performance patterns. This study demonstrates that an LLM's recall capability is not only contingent upon the prompt's content but also may be compromised by biases in its training data. Conversely, adjustments to model architecture, training strategy, or fine-tuning can improve performance. Our analysis provides insight into LLM behavior, offering direction for the development of more effective applications of LLMs.