# Plug 'n' Pray: Agentic LLM-based Detection of Potential Log File Exposures in Third-Party Content Management System Plugins

arXiv: https://arxiv.org/abs/2609.17164 · HTML: https://mlsanity.com/paper/2609.17164/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 55 references that are in the map.
- First posted on arXiv 15 Sep 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 11 of 11)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- [RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing](https://mlsanity.com/paper/2501.18160/) (2025)
- [Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection](https://mlsanity.com/paper/2408.12986/) (2024)
- [A Case Study of LLM for Automated Vulnerability Repair: Assessing Impact of Reasoning and Patch Validation Feedback](https://mlsanity.com/paper/2405.15690/) (2024)
- [CVE-LLM : Ontology-Assisted Automatic Vulnerability Evaluation Using Large Language Models](https://mlsanity.com/paper/2502.15932/) (2025)
- [What All the PHUZZ Is About: A Coverage-guided Fuzzer for Finding Vulnerabilities in PHP Web Applications](https://mlsanity.com/paper/2406.06261/) (2024)
- [Execution-State-Aware LLM Reasoning for Automated Proof-of-Vulnerability Generation](https://mlsanity.com/paper/2602.13574/) (2026)
- [A Demand-Side Viewpoint to Software Vulnerabilities in WordPress Plugins](https://mlsanity.com/paper/1812.05293/) (2018)
- [Over 100 Bugs in a Row: Security Analysis of the Top-Rated Joomla Extensions](https://mlsanity.com/paper/2102.03131/) (2021)

## Abstract (arXiv)

Content Management Systems (CMS), such as WordPress, power a large share of the web (~58%), and their extensibility through third-party plugins is a major source of their popularity as well as of their attack surface. One high-impact weakness that remains understudied is log file exposure by CMS plugins, which create log files for debugging or other purposes. If these files are insufficiently secured, they can disclose sensitive information (e.g. credentials, personal data) which has led to website compromises in the past. In this work, we present an agentic, LLM-based framework that automatically detects potential log file exposures in plugins of the most popular CMS (WordPress). Our agent analyzes each plugin by performing static and dynamic analysis. We evaluated our approach on the 300 most-installed WordPress plugins (about 0.6% of all), which together account for over 250M active installations, i.e. 75% of all active installations in the official plugin ecosystem. We manually validated each finding, reproducing 79 of 81 findings from 62 plugins. We observed that several protective measures appear to be implemented that we classify as creation-control (e.g. manual log activation) and access-control (e.g. deny rules in .htaccess). However, we find that multi-layered protection is required, but not always present. From these results we derive a taxonomy of log file path and protection patterns and deduce a set of best practices for developers to securely handle them. Finally, our study corroborates that agentic LLMs are an useful tool for security analysis.