Plug 'n' Pray: Agentic LLM-based Detection of Potential Log File Exposures in Third-Party Content Management System Plugins
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Content Management Systems (CMS), such as WordPress, power a large share of the web (~58%), and their extensibility through third-party plugins is a major source of their popularity as well as of their attack surface. One high-impact weakness that remains understudied is log file exposure by CMS plugins, which create log files for debugging or other purposes. If these files are insufficiently secured, they can disclose sensitive information (e.g. credentials, personal data) which has led to website compromises in the past. In this work, we present an agentic, LLM-based framework that automatically detects potential log file exposures in plugins of the most popular CMS (WordPress). Our agent analyzes each plugin by performing static and dynamic analysis. We evaluated our approach on the 300 most-installed WordPress plugins (about 0.6% of all), which together account for over 250M active installations, i.e. 75% of all active installations in the official plugin ecosystem. We manually validated each finding, reproducing 79 of 81 findings from 62 plugins. We observed that several protective measures appear to be implemented that we classify as creation-control (e.g. manual log activation) and access-control (e.g. deny rules in .htaccess). However, we find that multi-layered protection is required, but not always present. From these results we derive a taxonomy of log file path and protection patterns and deduce a set of best practices for developers to securely handle them. Finally, our study corroborates that agentic LLMs are an useful tool for security analysis.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 11 of its 55 references that are in the map. First posted on arXiv 15 Sep 2026.
Builds on
11 of 55 references.
Closest ancestors · 1
Same direction, newest first
Other references · 10
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 68
- 2025RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditingcited by 49
- 2024Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detectioncited by 28
- 2024A Case Study of LLM for Automated Vulnerability Repair: Assessing Impact of Reasoning and Patch Validation Feedbackcited by 22
- 2025CVE-LLM : Ontology-Assisted Automatic Vulnerability Evaluation Using Large Language Modelscited by 4
- 2024What All the PHUZZ Is About: A Coverage-guided Fuzzer for Finding Vulnerabilities in PHP Web Applicationscited by 3
- 2026Execution-State-Aware LLM Reasoning for Automated Proof-of-Vulnerability Generationcited by 3
- 2018A Demand-Side Viewpoint to Software Vulnerabilities in WordPress Pluginscited by 2
- 2021Over 100 Bugs in a Row: Security Analysis of the Top-Rated Joomla Extensionscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering
- 2026Indirect Prompt Injection in the Wild: An Empirical Study of Prevalence, Techniques, and Objectives
- 2026How Much Can We Trust LLM Search Agents? Measuring Endorsement Vulnerability to Web Content Manipulation
- 2025From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection
- 2026AgentRaft: Automated Detection of Data Over-Exposure in LLM Agents
- 2026LoginTrap: Uncovering Task-Agnostic Phishing-Style Indirect Prompt Injection Attacks against LLM-based Web Agents
- 2026Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer
- 2024Audit-LLM: Multi-Agent Collaboration for Log-based Insider Threat Detection