# FlexParser -- the adaptive log file parser for continuous results in a changing world

arXiv: https://arxiv.org/abs/2106.03170 · HTML: https://mlsanity.com/paper/2106.03170/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 45 references that are in the map.
- First posted on arXiv 6 Jun 2021.

## Builds on (most cited first, 9 of 9)

- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [LSTM Fully Convolutional Networks for Time Series Classification](https://mlsanity.com/paper/1709.05206/) (2017)
- [Tools and Benchmarks for Automated Log Parsing](https://mlsanity.com/paper/1811.03509/) (2018)
- [A Gentle Introduction to Deep Learning in Medical Image Processing](https://mlsanity.com/paper/1810.05401/) (2018)
- [Deep Gated Recurrent and Convolutional Network Hybrid Model for Univariate Time Series Classification](https://mlsanity.com/paper/1812.07683/) (2018)
- [Prediction of MRI Hardware Failures based on Image Features using Time Series Classification](https://mlsanity.com/paper/2001.02127/) (2020)

## Built on by (most cited first, 1 of 1)

- [Interactive Log Parsing via Light-weight User Feedback](https://mlsanity.com/paper/2301.12225/) (2023)

## Abstract (arXiv)

Any modern system writes events into files, called log files. Those contain crucial information which are subject to various analyses. Examples range from cybersecurity, intrusion detection over usage analyses to trouble shooting. Before data analysis is possible, desired information needs to be extracted first out of the semi-structured log messages. State-of-the-art event parsing often assumes static log events. However, any modern system is updated consistently and with updates also log file structures can change. We call those changes "mutation" and study parsing performance for different mutation cases. Latest research discovers mutations using anomaly detection post mortem, however, does not cover actual continuous parsing. Thus, we propose a novel and flexible parser, called FlexParser, which can extract desired values despite gradual changes in the log messages. It implies basic text preprocessing followed by a supervised Deep Learning method. We train a stateful LSTM on parsing one event per data set. Statefulness enforces the model to learn log message structures across several examples. Our model was tested on seven different, publicly available log file data sets and various kinds of mutations. Exhibiting an average F1-Score of 0.98, it outperforms other Deep Learning methods as well as state-of-the-art unsupervised parsers.