# Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs

arXiv: https://arxiv.org/abs/2604.20211 · HTML: https://mlsanity.com/paper/2604.20211/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 62 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Vulnerability Detection with Fine-grained Interpretations](https://mlsanity.com/paper/2106.10478/) (2021)
- [LILAC: Log Parsing using LLMs with Adaptive Parsing Cache](https://mlsanity.com/paper/2310.01796/) (2023)
- [A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?](https://mlsanity.com/paper/2308.10828/) (2023)
- [Multitask-based Evaluation of Open-Source LLM on Software Vulnerability](https://mlsanity.com/paper/2404.02056/) (2024)
- [ThinkRepair: Self-Directed Automated Program Repair](https://mlsanity.com/paper/2407.20898/) (2024)
- [What Makes a Good Commit Message?](https://mlsanity.com/paper/2202.02974/) (2022)
- [COFFE: A Code Efficiency Benchmark for Code Generation](https://mlsanity.com/paper/2502.02827/) (2025)
- [Did We Miss Something Important? Studying and Exploring Variable-Aware Log Abstraction](https://mlsanity.com/paper/2304.11391/) (2023)
- [Using Deep Learning to Generate Complete Log Statements](https://mlsanity.com/paper/2201.04837/) (2022)
- [Correctness Assessment of Code Generated by Large Language Models Using Internal Representations](https://mlsanity.com/paper/2501.12934/) (2025)
- [Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks](https://mlsanity.com/paper/2505.08903/) (2025)
- [SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios](https://mlsanity.com/paper/2509.22097/) (2025)
- [Go Static: Contextualized Logging Statement Generation](https://mlsanity.com/paper/2402.12958/) (2024)
- [Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM](https://mlsanity.com/paper/2501.14983/) (2025)
- [PEACE: Towards Efficient Project-Level Efficiency Optimization via Hybrid Code Editing](https://mlsanity.com/paper/2510.17142/) (2025)
- [Are They All Good? Studying Practitioners' Expectations on the Readability of Log Messages](https://mlsanity.com/paper/2308.08836/) (2023)
- [LogUpdater: Automated Detection and Repair of Specific Defects in Logging Statements](https://mlsanity.com/paper/2408.03101/) (2024)
- [Understanding Practitioners' Expectations on Clear Code Review Comments](https://mlsanity.com/paper/2410.06515/) (2024)
- [One-for-All Does Not Work! Enhancing Vulnerability Detection by Mixture-of-Experts (MoE)](https://mlsanity.com/paper/2501.16454/) (2025)
- [Protecting Privacy in Software Logs: What Should Be Anonymized?](https://mlsanity.com/paper/2409.11313/) (2024)
- [Studying and Benchmarking Large Language Models For Log Level Suggestion](https://mlsanity.com/paper/2410.08499/) (2024)
- [SDLog: A Deep Learning Framework for Detecting Sensitive Information in Software Logs](https://mlsanity.com/paper/2505.14976/) (2025)
- [Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning](https://mlsanity.com/paper/2508.11305/) (2025)
- [LLM4Perf: Large Language Models Are Effective Samplers for Multi-Objective Performance Modeling](https://mlsanity.com/paper/2512.16070/) (2025)

## Built on by (most cited first, 2 of 2)

- [SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements](https://mlsanity.com/paper/2607.27409/) (2026)
- [Towards Automatically Pruning Logging Code with Coding Agents: How Far Are We?](https://mlsanity.com/paper/2610.04716/) (2026)

## Abstract (arXiv)

Logging code plays an important role in software systems by recording key events and behaviors, which are essential for debugging and monitoring. However, insecure logging practices can inadvertently expose sensitive information or enable attacks such as log injection, posing serious threats to system security and privacy. Prior research has examined general defects in logging code, but systematic analysis of logging code security issues remains limited, particularly in leveraging LLMs for detection and repair. In this paper, we derive a comprehensive taxonomy of logging code security issues, encompassing four common issue categories and 10 corresponding patterns. We further construct a benchmark dataset with 101 real-world logging security issue reports that have been manually reviewed and annotated. We then propose an automated framework that incorporates various contextual knowledge to evaluate LLMs' capabilities in detecting and repairing logging security issues. Our experimental results reveal a notable disparity in performance: while LLMs are moderately effective at detecting security issues (e.g., the accuracy ranges from 12.9% to 52.5% on average), they face noticeable challenges in reliably generating correct code repairs. We also find that the issue description alone improves the LLMs' detection accuracy more than the security pattern explanation or a combination of both. Overall, our findings provide actionable insights for practitioners and highlight the potential and limitations of current LLMs for secure logging.