# InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information

arXiv: https://arxiv.org/abs/2608.29632 · HTML: https://mlsanity.com/paper/2608.29632/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 30 Aug 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 20 of 20)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings](https://mlsanity.com/paper/2501.01257/) (2025)
- [Can Language Models Solve Olympiad Programming?](https://mlsanity.com/paper/2404.10952/) (2024)
- [LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?](https://mlsanity.com/paper/2506.11928/) (2025)
- [CodeContests+: High-Quality Test Case Generation for Competitive Programming](https://mlsanity.com/paper/2506.05817/) (2025)
- [OJBench: A Competition Level Code Benchmark For Large Language Models](https://mlsanity.com/paper/2506.16395/) (2025)
- [HardTests: Synthesizing High-Quality Test Cases for LLM Coding](https://mlsanity.com/paper/2505.24098/) (2025)
- [ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests](https://mlsanity.com/paper/2506.04894/) (2025)
- [AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions](https://mlsanity.com/paper/2508.16402/) (2025)
- [Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?](https://mlsanity.com/paper/2506.12713/) (2025)
- [LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?](https://mlsanity.com/paper/2510.09595/) (2025)

## Abstract (arXiv)

Competitive programming is increasingly being used to evaluate the algorithmic reasoning capabilities of large language models (LLMs). However, existing benchmarks primarily focus on full-information tasks where all problem inputs are provided upfront. This overlooks a critical dimension of algorithmic reasoning: the ability of generated programs to operate when key information is not revealed upfront. Interactive problems, a distinctive component of competitive programming, embody this challenge. These problems require programs to engage in multi-round interaction with an interactor (a judge program) under strict protocol constraints and limited query budgets, with new information revealed only in response to queries. To address this gap, we introduce InteractBench, a benchmark comprising 322 high-quality interactive problems curated from Codeforces, AtCoder, IOI, and ICPC. Each problem is packaged with executable local interactors, enabling fully offline evaluation. Unlike existing benchmarks, InteractBench assesses whether model-generated code can acquire information and track state dynamically. Our evaluation reveals a significant interaction gap: even the most advanced reasoning models achieve limited success on interactive problems. Beyond success rates, we propose a fine-grained failure taxonomy to diagnose the root causes of these deficiencies. Although algorithmic logic errors remain dominant, protocol violations and query-budget overruns are frequent. Code is available at https://github.com/kmsgk0/InteractBench.