mlsanityData to 7 Oct 2026

Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts

arXiv 2609.18366 ↗First posted 16 Sep 2026cs.AI cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark Brel to guide a Proposer that edits prompts, memory, retrieval, tools, and control code around a fixed foundation model. Task holdout is commonly used to guard against harness overfitting. It varies semantic tasks but leaves the benchmark protocol fixed, so a bad genius Proposer can produce a cheating harness whose improvement over the initial harness on Brel depends on a benchmark-wide shortcut. We introduce Counterfactual Harness Search and Evolution (CHASE), which casts harness evolution as constraint generation over valid counterfactual benchmarks. After each Proposer update, a Challenger searches for an executable protocol transformation with large gain destruction. A validity firewall checks that task semantics are preserved, while a held-out confirmation set determines whether the counterfactual enters a finite archive. We formalize an ideal shortcut-neutralized benchmark B0 and establish theoretical guarantees linking finite counterfactual archives to B0 and characterizing sequential Challenger search. We evaluate CHASE on Syn-Ledger and OfficeQA, where CHASE retains strong released-benchmark gains while substantially reducing gain destruction under valid protocol transformations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 46 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

42 of 46 references.

References · 42

  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  2. 2022Red Teaming Language Models with Language Modelscited by 983
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  4. 2023Large Language Models as Optimizerscited by 830
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  7. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  8. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  9. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  10. 2026EvoSkill: Automated Skill Discovery for Multi-Agent Systemscited by 96
  11. 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflowscited by 44
  12. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 42
30 more
  1. 2026Rethinking the Evaluation of Harness Evolution for Agentscited by 39
  2. 2026OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoningcited by 37
  3. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  4. 2023METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualitiescited by 20
  5. 2026HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnessescited by 13
  6. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  7. 2026BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluationcited by 12
  8. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  9. 2026AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Tracescited by 9
  10. 2026Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluationcited by 8
  11. 2026EnvHarness: Awakening Static Worlds for Agent Learningcited by 7
  12. 2026Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verificationcited by 7
  13. 2021SpecRepair: Counter-Example Guided Safety Repair of Deep Neural Networkscited by 6
  14. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 6
  15. 2026Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AIcited by 6
  16. 2026HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolutioncited by 6
  17. 2026HarnessOpt-Bench: Evaluating LLMs at Harness Optimizationcited by 5
  18. 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilitiescited by 4
  19. 2023A Robust Optimisation Perspective on Counterexample-Guided Repair of Neural Networkscited by 3
  20. 2026"LLM Agent Performance" Is Not a Single Evaluation Targetcited by 3
  21. 2026How Benchmarks Mis-Score Computer-Use Agentscited by 3
  22. 2026Auditing Harness Tampering in Self-Improving Agentscited by 3
  23. 2026Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnessescited by 3
  24. 2026Towards Direct Evaluation of Harness Optimizers via Priority Rankingcited by 2
  25. 2026Harness Continual Learning: Continual Adaptation Beyond Model Parameterscited by 2
  26. 2026EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agentscited by 2
  27. 2026Chaos Is a LADDER: Domain Generalization Beyond Invariance via Reweightingcited by 1
  28. 2026StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environmentscited by 1
  29. 2026Harness-agnostic detection and immunization of reward hacking in self-evolving language modelscited by 1
  30. 2026A Theory of Reliable Self-Evolution for Agent Harnessescited by 1

Built on by

Not cited yet.