mlsanityData to 7 Oct 2026

OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding

arXiv 2601.10343 ↗First posted 15 Jan 2026cs.CL cs.AIcited by 15

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Modern coding scaffolds turn LLMs into capable software agents, but their ability to follow scaffold-specified instructions remains under-examined, especially when constraints are heterogeneous and persist across interactions. To fill this gap, we introduce OctoBench, which benchmarks scaffold-aware instruction following in repository-grounded agentic coding. OctoBench includes 34 environments and 217 tasks instantiated under three scaffold types, and is paired with 7,098 objective checklist items. To disentangle solving the task from following the rules, we provide an automated observation-and-scoring toolkit that captures full trajectories and performs fine-grained checks. Experiments on eight representative models reveal a systematic gap between task-solving and scaffold-aware compliance, underscoring the need for training and evaluation that explicitly targets heterogeneous instruction following. We release the benchmark to support reproducible benchmarking and to accelerate the development of more scaffold-aware coding agents.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 18 of its 39 references that are in the map. First posted on arXiv 15 Jan 2026.

Builds on

18 of 39 references.

References · 18

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  9. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  10. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  11. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  12. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
6 more
  1. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 48
  2. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 48
  3. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 44
  4. 2024NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctnesscited by 23
  5. 2025CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generationcited by 18
  6. 2025SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositoriescited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026Natural-Language Agent Harnessescited by 54
  2. 2026CL-bench Life: Can Language Models Learn from Real-Life Context?cited by 8
  3. 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actionscited by 6
  4. 2026EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Trainingcited by 6
  5. 2026The Verification Horizon: No Silver Bullet for Coding Agent Rewardscited by 6
  6. 2026Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?cited by 5
  7. 2026Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operationscited by 4
  8. 2026Does Pass Rate Tell the Whole Story? Evaluating Design Constraint Compliance in LLM-based Issue Resolutioncited by 2
  9. 2026FAVA: Formal Authorization for Verified Agents with Evidence-Backed Permission Graphscited by 2
  10. 2026Agents in the Large: Perception-Centered Architecture for Persistent Agentscited by 1
  11. 2026A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communitiescited by 0
  12. 2026QuoteBench: How Matched Scores Can Hide Command-Path Failurescited by 0
3 more
  1. 2026Terminal Agents: A Survey of AI Agents in Command-Line Environmentscited by 0
  2. 2026The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?cited by 0
  3. 2026MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Codingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Schr"odinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?
  2. 2026Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures
  3. 2026When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents
  4. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding
  5. 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
  6. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
  7. 2026Steerable Instruction Following Coding Data Synthesis with Actor-Parametric Schema Co-Evolution
  8. 2026LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation