mlsanityData to 7 Oct 2026

LogDx-CI: Benchmarking Log Reduction Tools for LLM Root-Cause Diagnosis

arXiv 2605.28876 ↗First posted 26 May 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

CI failure logs are large (median 5k lines, max 200k in this corpus) and noisy. Coding agents that try to debug them depend on an upstream tool to reduce the log to a manageable context, but the field has had no public empirical comparison of which reductions preserve enough evidence for downstream LLM diagnosis. We introduce LogDx-CI, a benchmark that compares 11 context-reduction tools (raw, tail, grep, three RTK modes, two real LLM map-reduce summarizers, three hybrid routers) on 35 real GitHub Actions failure cases, scored by 3 LLM debugger families (Claude Haiku 4.5, Claude Sonnet 4.6, OpenAI gpt-5-mini) plus a Sonnet 4.6 tool-using agent. We report three load-bearing findings. (1)~Hybrid grep+tail routers dominate the cost-quality Pareto frontier; the top two methods score 0.670 / 0.666 at ~ 0.03 per case, same-ballpark quality as standalone grep at 4.5× fewer tokens. (2)~In the agent-loop regime, the quality range across reduction tools collapses 7× (single-shot spread 0.42 → agent-loop spread 0.059); the agent rescues weak contexts via follow-up tool calls. However, cost differences persist: weak contexts force the agent to issue 2--4× more tool calls to recover. (3)~A cross-family LLM-summary pair (gpt-5-mini summarizer feeding a Claude Haiku debugger) beats the same-family pair by +0.071 averaged across four diagnoser variants, falsifying the self-call-bias hypothesis on this task. The gpt-5-mini summarizer is also the agent-loop #1 method (score 0.749) at 0.37 tool-calls per case and 10× lower reducer cost than the Haiku summarizer (0.18 vs 1.75 per case). All data, code, per-case bundles, and reproducibility infrastructure are public.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 9 of its 15 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

9 of 15 references.

References · 9

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  7. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  8. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  9. 2018Tools and Benchmarks for Automated Log Parsingcited by 56

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LogSieve: Task-Aware CI Log Reduction for Sustainable LLM-Based Analysis
  2. 2026FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets
  3. 2025LogSage: An LLM-Based Framework for CI/CD Failure Detection and Remediation with Industrial Validation
  4. 2026AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
  5. 2025Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning
  6. 2026TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps
  7. 2026Ekka: Automated Diagnosis of Silent Errors in LLM Inference
  8. 2026CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend