mlsanityData to 7 Oct 2026

A Case Study of Web App Coding with OpenAI Reasoning Models

arXiv 2409.13773 ↗First posted 19 Sep 2024cs.SE cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

This paper presents a case study of coding tasks by the latest reasoning models of OpenAI, i.e. o1-preview and o1-mini, in comparison with other frontier models. The o1 models deliver SOTA results for WebApp1K, a single-task benchmark. To this end, we introduce WebApp1K-Duo, a harder benchmark doubling number of tasks and test cases. The new benchmark causes the o1 model performances to decline significantly, falling behind Claude 3.5. Moreover, they consistently fail when confronted with atypical yet correct test cases, a trap non-reasoning models occasionally avoid. We hypothesize that the performance variability is due to instruction comprehension. Specifically, the reasoning mechanism boosts performance when all expectations are captured, meanwhile exacerbates errors when key expectations are missed, potentially impacted by input lengths. As such, we argue that the coding success of reasoning models hinges on the top-notch base model and SFT to ensure meticulous adherence to instructions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 13 of its 23 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

13 of 23 references.

References · 13

  1. 2023Let's Verify Step by Stepcited by 3,598
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  5. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  7. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  8. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  9. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  10. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  11. 2024WebApp1K: A Practical Code-Generation Benchmark for Web App Developmentcited by 10
  12. 2023Action Inference by Maximising Evidence: Zero-Shot Imitation from Observation with World Modelscited by 5
1 more
  1. 2024Insights from Benchmarking Frontier Language Models on Web App Code Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Model
  2. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
  3. 2026Codoku: Renewable Program-Reasoning Challenges for Frontier Coding Agents
  4. 2025The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Models
  5. 2026Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks
  6. 2026Unified Deployment-Aware Evaluation of Open Reasoning Language Models
  7. 2026Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement
  8. 2025Reasoning as a Resource: Optimizing Fast and Slow Thinking in Code Generation Models