mlsanityData to 14 Oct 2026

Audit tools of the trade: Automated Auditor Elicitation Can Be Sensitive to the Context the Auditor Is Given

arXiv 2610.16911 ↗First posted 12 Oct 2026cs.AIcited by 0

New in LLM alignment / RLHF / preference optimisation; no direction yet. LLM alignment / RLHF / preference optimisation

Abstract (arXiv)

Does providing additional context to automated AI auditors improve their effectiveness? Using the Petri tool to audit LLM coding agents, we test the impact of providing two types of resources (an external codebase and benchmark transcripts) to LLM auditors, across four auditor-target LLM pairs and three behaviours (harmful compliance, reward hacking, sycophancy). (1) External codebase: We provide the auditor LLM with a fixed, external open source codebase, which prior work finds improves the realism of synthetic audits. Providing this resource suppresses reward hacking in both auditor-target pairs whose baseline rate is above the floor (4.8 to 1.3 and 3.9 to 1.8 on Petri's default 1-10 scale); the remaining two pairs sit at the floor throughout. The suppression recurs in three of four pairs under a different scenario and resource bundle. (2) External benchmark transcripts: We provide the auditor with transcripts of the target's performance on benchmarks measuring similar behaviours, intended to inform the auditor about the target's behavioural patterns. Providing this resource raises reward hacking scores in all four pairs (from 1.7 to 6.1 in one case). However, subsequent experiments show that supplying only the related benchmark's abstract, without transcripts, also increases reward hacking, and the effect is absent under a different framing of the same behaviour. Overall, we find no consistent effect of either resource outside reward hacking, but we find substantial variability in judge-scored behaviour across conditions. Our results underscore the importance of careful piloting with tools such as Petri, as the measured rate of target behaviour is highly sensitive to configuration decisions. We hope our results inspire more research into the reliability and robustness of automated alignment auditing.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 32 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

18 of 32 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  5. 2022Red Teaming Language Models with Language Modelscited by 1,026
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  8. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  9. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  10. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  11. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  12. 2025Auditing language models for hidden objectivescited by 83
6 more
  1. 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 61
  2. 2026Evaluating whether AI models would sabotage AI safety researchcited by 10
  3. 2026Predicting LLM Safety Before Release by Simulating Deploymentcited by 8
  4. 2026How Inference Compute Shapes Frontier LLM Evaluationcited by 7
  5. 2026An Independent Safety Evaluation of Kimi K2.5cited by 3
  6. 2026Pressure Reveals Character: Behavioural Alignment Evaluation at Depthcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases
  2. 2026Demystifying Agent Skills for Smart Contract Auditing: Design, Effectiveness, Behavioral Impact
  3. 2026When and How Context Rot Appears in Coding Agents: A White-Box Study of Agent Skills in Code Auditing
  4. 2026Automated Benchmark Auditing for AI Agents and Large Language Models
  5. 2026AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
  6. 2026Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing
  7. 2026Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research
  8. 2026Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems