# DENIAHL: In-Context Features Influence LLM Needle-In-A-Haystack Abilities

arXiv: https://arxiv.org/abs/2411.19360 · HTML: https://mlsanity.com/paper/2411.19360/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2024.

## Built on by (most cited first, 3 of 3)

- [LongFuncEval: Measuring the effectiveness of long context models for function calling](https://mlsanity.com/paper/2505.10570/) (2025)
- [Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments](https://mlsanity.com/paper/2602.16653/) (2026)
- [Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench](https://mlsanity.com/paper/2607.08284/) (2026)

## Abstract (arXiv)

The Needle-in-a-haystack (NIAH) test is a general task used to assess language models' (LMs') abilities to recall particular information from long input context. This framework however does not provide a means of analyzing what factors, beyond context length, contribute to LMs' abilities or inabilities to separate and recall needles from their haystacks. To provide a systematic means of assessing what features contribute to LMs' NIAH capabilities, we developed a synthetic benchmark called DENIAHL (Data-oriented Evaluation of NIAH for LLM's). Our work expands on previous NIAH studies by ablating NIAH features beyond typical context length including data type, size, and patterns. We find stark differences between GPT-3.5 and LLaMA 2-7B's performance on DENIAHL, and drops in recall performance when features like item size are increased, and to some degree when data type is changed from numbers to letters. This has implications for increasingly large context models, demonstrating factors beyond item-number impact NIAH capabilities.