# textttYC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Execution

arXiv: https://arxiv.org/abs/2604.01212 · HTML: https://mlsanity.com/paper/2604.01212/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 11 of 11)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents](https://mlsanity.com/paper/2401.10568/) (2024)

## Built on by (most cited first, 11 of 11)

- [FutureSim: Replaying World Events to Evaluate Adaptive Agents](https://mlsanity.com/paper/2605.15188/) (2026)
- [CEO-Bench: Can Agents Play the Long Game?](https://mlsanity.com/paper/2606.18543/) (2026)
- [Benchmarking Open-Ended Multi-Agent Coordination in Language Agents](https://mlsanity.com/paper/2606.08340/) (2026)
- [Business Arena: Benchmarking LLM Agents in a Realistic Marketplace](https://mlsanity.com/paper/2608.08621/) (2026)
- [Apodex 1.1: Scaling Agentic Intelligence for Complex Work](https://mlsanity.com/paper/2608.23283/) (2026)
- [E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation](https://mlsanity.com/paper/2608.30730/) (2026)
- [FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents](https://mlsanity.com/paper/2608.18423/) (2026)
- [LEAP: Trajectory-Level Evaluation of LLMs in Iterative Scientific Design](https://mlsanity.com/paper/2605.15341/) (2026)
- [OpenLoopEvolve: A Verifiable Self-Evolution Framework for Loop Policies in Long-Horizon Complex Tasks](https://mlsanity.com/paper/2608.09380/) (2026)
- [FromPitch2Board: Benchmarking LLM Agents in Long-Horizon Football Management](https://mlsanity.com/paper/2609.34710/) (2026)
- [AgentTime: Can Agents Estimate and Control Their Own Runtime?](https://mlsanity.com/paper/2610.09944/) (2026)

## Abstract (arXiv)

As LLM agents tackle increasingly complex tasks, a critical question is whether they can maintain strategic coherence over long horizons: planning under uncertainty, learning from delayed feedback, and adapting when early mistakes compound. We introduce textttYC-Bench, a benchmark that evaluates these capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns. The agent must manage employees, select task contracts, and maintain profitability in a partially observable environment where adversarial clients and growing payroll create compounding consequences for poor decisions. We evaluate 12 models, both proprietary and open source, across 3 seeds each. Only three models consistently surpass the starting capital of 200K, with Claude Opus 4.6 achieving the highest average final funds at 1.27 M, followed by GLM-5 at 1.21 M at 11× lower inference cost. Scratchpad usage, the sole mechanism for persisting information across context truncation, is the strongest predictor of success, and adversarial client detection is the primary failure mode, accounting for 47% of bankruptcies. Our analysis reveals that frontier models still fail through distinct failure modes such as over-parallelization, demonstrating the capability gaps for long-horizon performance. textttYC-Bench is open-source, reproducible, and configurable.