# Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answering

arXiv: https://arxiv.org/abs/2602.17183 · HTML: https://mlsanity.com/paper/2602.17183/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 3 of its 10 references that are in the map.
- First posted on arXiv 19 Feb 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 3 of 3)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [LongCodeBench: Evaluating Coding LLMs at 1M Context Windows](https://mlsanity.com/paper/2505.07897/) (2025)
- [LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding](https://mlsanity.com/paper/2503.04359/) (2025)

## Built on by (most cited first, 1 of 1)

- [Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study](https://mlsanity.com/paper/2608.01507/) (2026)

## Abstract (arXiv)

Large language models (LLMs) increasingly assist software engineering tasks that require reasoning over long code contexts, yet their robustness under varying input conditions remains unclear. We conduct a systematic study of long-context code question answering using controlled ablations that test sensitivity to answer format, distractors, and context scale. Extending LongCodeBench Python dataset with new COBOL and Java question-answer sets, we evaluate state-of-the-art models under three settings: (i) shuffled multiple-choice options, (ii) open-ended questions and (iii) needle-in-a-haystack contexts containing relevant and adversarially irrelevant information. Results show substantial performance drops in both shuffled multiple-choice options and open-ended questions, and brittle behavior in the presence of irrelevant cues. Our findings highlight limitations of current long-context evaluations and provide a broader benchmark for assessing code reasoning in both legacy and modern systems.