Qiushi Engine on AstaBench E2E-Bench-Hard
Not on the map. Why?
Abstract (arXiv)
This report analyzes Qiushi Engine v0.8 across all 40 test tasks in AstaBench E2E-Bench-Hard, a benchmark that requires autonomous agents to carry a research question through experimental design, code implementation, actual execution, result analysis, and report delivery. Qiushi Engine is model-configurable; this evaluation selected DeepSeek deepseek-v4pro-preview as the model backend. The official AstaBench leaderboard records a score of 0.816 and an average benchmark cost of USD 15.209 per task, while the full-precision local recomputation is 81.59 ± 1.87. Four tasks satisfied every rubric item, yielding a full-task completion rate of 4/40 = 10% -- 7 percentage points above, and about 3.3 times, the approximately 3% best rate reported for AstaBench's official agents. Across 507 required rubric items, 416 were satisfied (82.1%). Official scoring archives and 40 Meta-Trace records show sustained production and verification of reports, code, and experimental artifacts; the principal gaps lie in repeated runs, external dependencies, specified metrics, and ablation studies. The report explains the benchmark, system workflow, aggregate results, representative cases, and limits of interpretation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 5 of its 13 references that are in the map. First posted on arXiv 8 Sep 2026.
Builds on
5 of 13 references.
References · 5
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
- 2025AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suitecited by 45
Built on by
Not cited yet.