# Chatting with Logs: An exploratory study on Finetuning LLMs for LogQL

arXiv: https://arxiv.org/abs/2412.03612 · HTML: https://mlsanity.com/paper/2412.03612/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 50 references that are in the map.
- First posted on arXiv 4 Dec 2024.

## Builds on (most cited first, 15 of 15)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://mlsanity.com/paper/2008.06448/) (2020)
- [FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservices](https://mlsanity.com/paper/2008.08509/) (2020)
- [LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing](https://mlsanity.com/paper/2404.18001/) (2024)
- [Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study](https://mlsanity.com/paper/2404.17136/) (2024)
- [AI-Assisted SQL Authoring at Industry Scale](https://mlsanity.com/paper/2407.13280/) (2024)
- [Enhancing LLM Fine-tuning for Text-to-SQLs by SQL Quality Measurement](https://mlsanity.com/paper/2410.01869/) (2024)
- [Positional Paper: Schema-First Application Telemetry](https://mlsanity.com/paper/2206.11380/) (2022)

## Built on by (most cited first, 1 of 1)

- [Statistical Guarantees in Synthetic Data through Conformal Adversarial Generation](https://mlsanity.com/paper/2504.17058/) (2025)

## Abstract (arXiv)

Logging is a critical function in modern distributed applications, but the lack of standardization in log query languages and formats creates significant challenges. Developers currently must write ad hoc queries in platform-specific languages, requiring expertise in both the query language and application-specific log details -- an impractical expectation given the variety of platforms and volume of logs and applications. While generating these queries with large language models (LLMs) seems intuitive, we show that current LLMs struggle with log-specific query generation due to the lack of exposure to domain-specific knowledge. We propose a novel natural language (NL) interface to address these inconsistencies and aide log query generation, enabling developers to create queries in a target log query language by providing NL inputs. We further introduce ~NL2QL, a manually annotated, real-world dataset of natural language questions paired with corresponding LogQL queries spread across three log formats, to promote the training and evaluation of NL-to-loq query systems. Using NL2QL, we subsequently fine-tune and evaluate several state of the art LLMs, and demonstrate their improved capability to generate accurate LogQL queries. We perform further ablation studies to demonstrate the effect of additional training data, and the transferability across different log formats. In our experiments, we find up to 75% improvement of finetuned models to generate LogQL queries compared to non finetuned models.