# Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge

arXiv: https://arxiv.org/abs/2605.08518 · HTML: https://mlsanity.com/paper/2605.08518/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 16 of 16)

- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.00212/) (2025)
- [PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition](https://mlsanity.com/paper/2407.11214/) (2024)
- [CRAG -- Comprehensive RAG Benchmark](https://mlsanity.com/paper/2406.04744/) (2024)
- [Codabench: Flexible, Easy-to-Use and Reproducible Benchmarking Platform](https://mlsanity.com/paper/2110.05802/) (2021)
- [Learning to run a Power Network Challenge: a Retrospective Analysis](https://mlsanity.com/paper/2103.03104/) (2021)
- [Results of the Big ANN: NeurIPS'23 competition](https://mlsanity.com/paper/2409.17424/) (2024)
- [Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition](https://mlsanity.com/paper/2406.07954/) (2024)
- [AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance](https://mlsanity.com/paper/2506.03828/) (2025)
- [General Agent Evaluation](https://mlsanity.com/paper/2602.22953/) (2026)
- [How Reliable is Language Model Micro-Benchmarking?](https://mlsanity.com/paper/2510.08730/) (2025)
- [Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos](https://mlsanity.com/paper/2407.09100/) (2024)
- [FAIR Universe HiggsML Uncertainty Dataset and Competition](https://mlsanity.com/paper/2410.02867/) (2024)
- [MedAI: Evaluating TxAgent's Therapeutic Agentic Reasoning in the NeurIPS CURE-Bench Competition](https://mlsanity.com/paper/2512.11682/) (2025)

## Built on by (most cited first, 2 of 2)

- [Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents](https://mlsanity.com/paper/2606.19704/) (2026)
- [Task-Aware Harness Provisioning for LLM Agents in Mission-Critical Infrastructure Operations](https://mlsanity.com/paper/2608.17433/) (2026)

## Abstract (arXiv)

Competition retrospectives are useful when they explain what a leaderboard measured, how hidden evaluation changed conclusions, and which design patterns were rewarded. We revisit the CODS 2025 assetopslive challenge, a privacy-aware Codabench competition on industrial multi-agent orchestration built on assetops. We combine final rank sheets, a 300-submission server log, 149-team registrations, best-submission exports, the organizer winners report, the companion assetopslive system paper, and verified planning-track source trees. Five results stand out. First, the public planning leaderboard saturates at 72.73%, and richer prompts do not improve that peak. Second, hidden evaluation changes the story: public and private scores correlate moderately in planning (r=0.69) but negatively in execution (r=-0.13), with several 45.45% public execution systems reaching 63.64% on the hidden set. Third, the tmatch term is numerically almost inert in the official composite -- combined on a 0--1 scale with 0--100 percentage scores, it contributes at most 0.05 points per track, and rescaling would swap the top two teams. Fourth, the competition is operationally account-based but substantively team-based: 149 registered teams reduce to 24 with non-zero public scores and 11 fully ranked, while 52.3% of deduplicated registrations list multiple usernames. Fifth, successful execution methods mostly improve guardrails -- response selection, contamination cleanup, fallback, and context control -- rather than novel agent architectures. These findings identify which behaviors the evaluation rewarded, and motivate scale-aware composites, skill-level diagnostics, and versioned artifact release.