mlsanityData to 7 Oct 2026

Seven simple steps for log analysis in AI systems

arXiv 2604.09563 ↗First posted 13 Feb 2026cs.AI cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

AI systems produce large volumes of logs as they interact with tools and users. Analysing these logs can help understand model capabilities, propensities, and behaviours, or assess whether an evaluation worked as intended. Researchers have started developing methods for log analysis, but a standardised approach is still missing. Here we suggest a pipeline based on current best practices. We illustrate it with concrete code examples in the Inspect Scout library, provide detailed guidance on each step, and highlight common pitfalls. Our framework provides researchers with a foundation for rigorous and reproducible log analysis.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 13 Feb 2026.

Builds on

31 of 50 references.

References · 31

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  7. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  8. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  10. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  11. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  12. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
19 more
  1. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  2. 2024On Verbalized Confidence Scores for LLMscited by 83
  3. 2025Great Models Think Alike and this Undermines AI Oversightcited by 53
  4. 2025Investigating Affective Use and Emotional Well-being on ChatGPTcited by 36
  5. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  6. 2023DialogBench: Evaluating LLMs as Human-like Dialogue Systemscited by 31
  7. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 30
  8. 2024Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Modelscited by 27
  9. 2025Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Modelscited by 25
  10. 2025How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Studycited by 21
  11. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 21
  12. 2024Re-evaluating Automatic LLM System Ranking for Alignment with Human Preferencecited by 16
  13. 2025Lessons from a Chimp: AI "Scheming" and the Quest for Ape Languagecited by 16
  14. 2025HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statisticscited by 14
  15. 2025Large Language Models Do Multi-Label Classification Differentlycited by 12
  16. 2024Conversational Complexity for Assessing Risk in Large Language Modelscited by 6
  17. 2025Skewed Score: A statistical framework to assess autograderscited by 6
  18. 2025People readily follow personal advice from AI but it does not improve their well-beingcited by 6
  19. 2025LM Fight Arena: Benchmarking Large Multimodal Models via Game Competitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Log analysis is necessary for credible evaluation of AI agentscited by 8
  2. 2026Open-World Evaluations for Measuring Frontier AI Capabilitiescited by 3
  3. 2026Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarkscited by 1
  4. 2026Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluationscited by 0
  5. 2026TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Tracescited by 0