mlsanityData to 7 Oct 2026

AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor

arXiv 2601.05752 ↗First posted 9 Jan 2026cs.CL cs.SEcited by 8

Not on the map. Why?

Abstract (arXiv)

We introduce AutoMonitor-Bench, the first benchmark designed to systematically evaluate the reliability of LLM-based misbehavior monitors across diverse tasks and failure modes. AutoMonitor-Bench consists of 3,010 carefully annotated test samples spanning question answering, code generation, and reasoning, with paired misbehavior and benign instances. We evaluate monitors using two complementary metrics: Miss Rate (MR) and False Alarm Rate (FAR), capturing failures to detect misbehavior and oversensitivity to benign behavior, respectively. Evaluating 12 proprietary and 10 open-source LLMs, we observe substantial variability in monitoring performance and a consistent trade-off between MR and FAR, revealing an inherent safety-utility tension. To further explore the limits of monitor reliability, we construct a large-scale training corpus of 153,581 samples and fine-tune Qwen3-4B-Instruction to investigate whether training on known, relatively easy-to-construct misbehavior datasets improves monitoring performance on unseen and more implicit misbehaviors. Our results highlight the challenges of reliable, scalable misbehavior monitoring and motivate future work on task-aware designing and training strategies for LLM-based monitors.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 9 Jan 2026.

Builds on

29 of 58 references.

References · 29

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  9. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  10. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  11. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  12. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
17 more
  1. 2025HalluLens: LLM Hallucination Benchmarkcited by 88
  2. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  3. 2025Understanding Aha Moments: from External Observations to Internal Mechanismscited by 48
  4. 2023Fake Alignment: Are LLMs Really Aligned Well?cited by 39
  5. 2024BEARS Make Neuro-Symbolic Models Aware of their Reasoning Shortcutscited by 36
  6. 2025Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforcecited by 33
  7. 2025Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducementscited by 31
  8. 2024Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScorecited by 27
  9. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 20
  10. 2025Model-agnostic Adversarial Attack and Defense for Vision-Language-Action Modelscited by 19
  11. 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMscited by 18
  12. 2025Goal-oriented Backdoor Attack against Vision-Language-Action Models via Physical Objectscited by 16
  13. 2025Is Your LLM-Based Multi-Agent a Reliable Real-World Planner? Exploring Fraud Detection in Travel Planningcited by 14
  14. 2025Can Large Language Models Identify Implicit Suicidal Ideation? An Empirical Evaluationcited by 13
  15. 2025LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancementscited by 8
  16. 2025DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Responsecited by 4
  17. 2025The Compositional Architecture of Regret in Large Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stabilitycited by 4
  2. 2026DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detectioncited by 3
  3. 2026PreAct-Bench: Benchmarking Predictive Monitoring in LLMscited by 1
  4. 2026Reward Hacking Challenges Oversight of Autonomous Research Agentscited by 1
  5. 2026CoT-Guard: Small Models for Strong Monitoringcited by 0
  6. 2026Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarkscited by 0
  7. 2026Silence Is Endorsement: Verification-Status Laundering in LLM Agent Pipelinescited by 0
  8. 2026RH-Detect: A Unified Benchmark for Reward Hacking Detectioncited by 0