mlsanityData to 7 Oct 2026

Benchmarking Mythos-Linked Bug Rediscovery

arXiv 2605.17416 ↗First posted 17 May 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Anthropic's April 2026 Mythos materials combine benchmark claims with concrete bug-finding stories across OpenBSD, FreeBSD, Linux, FFmpeg, and browsers. This paper reports a controlled target-file rediscovery experiment on six public or high-confidence Mythos-linked systems tasks. Each model receives the same target file or files, read-only source tools, three repeats per task, and one manual target-matching rubric; prompts omit CVE identifiers, patch hashes, advisory text, author names, disclosure dates, and answer key root cause language. The experiment contains 54 counted model-task attempts: three models, six tasks, and three repeats, giving 18 attempts per model. GPT-5.5 xhigh achieves 5/18 target rediscoveries, covering 2/6 tasks; counting one wrong-target mpegts.c finding separately gives 3/6 distinct core bugs. Claude Opus 4.7 achieves 1/18 target rediscoveries, covering 1/6 tasks. Kimi K2 records 0/18 target rediscoveries. The dominant failure mode is early commitment to plausible alternate candidates within the assigned file: models often submit source-grounded hypotheses while missing the specific invariant corrected by public Mythos patch evidence. These results do not refute Anthropic's undisclosed workflow, but show that under this favorable target-file scaffold, systems-specific prompting yields only six target matches across 54 counted attempts.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 41 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

20 of 41 references.

References · 20

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  5. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  6. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  7. 2024LLM Agents can Autonomously Hack Websitescited by 108
  8. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
  9. 2023PentestGPT: An LLM-empowered Automatic Penetration Testing Toolcited by 94
  10. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  11. 2018SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilitiescited by 91
  12. 2021CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Softwarecited by 89
8 more
  1. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 84
  2. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testingcited by 35
  3. 2019A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Softwarecited by 28
  4. 2025Multi-Agent Penetration Testing AI for the Webcited by 19
  5. 2024Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Studycited by 17
  6. 2025AI Agent Smart Contract Exploit Generationcited by 8
  7. 2024How Well Do Large Language Models Serve as End-to-End Secure Code Agents for Python?cited by 7
  8. 2026Towards Optimal Agentic Architectures for Offensive Security Taskscited by 4

Built on by

Not cited yet.