# Exploring Block Anomaly Detection In HDFS Log Data Analysis

arXiv: https://arxiv.org/abs/2607.29383 · HTML: https://mlsanity.com/paper/2607.29383/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://mlsanity.com/paper/2008.06448/) (2020)
- [LAnoBERT: System Log Anomaly Detection based on BERT Masked Language Model](https://mlsanity.com/paper/2111.09564/) (2021)
- [Deep Learning-based Anomaly Detection and Log Analysis for Computer Networks](https://mlsanity.com/paper/2407.05639/) (2024)
- [Linq-Embed-Mistral Technical Report](https://mlsanity.com/paper/2412.03223/) (2024)
- [A Comprehensive Survey of Logging in Software: From Logging Statements Automation to Log Mining and Analysis](https://mlsanity.com/paper/2110.12489/) (2021)

## Abstract (arXiv)

In recent years, with the development of big data technology, increasingly more companies use HDFS for data processing and storage. As a result, the maintenance of distributed file systems has become an extremely important part of data management. As the function of server systems is becoming increasingly diversified and their services are becoming complex, the logs, recording real-time events make it easier for system operators to locate the failures and errors that happened in the server systems to make server always available. HDFS, a distributed file system, which contains large data sets, will record a large number of logs. Moreover, the logs are not always structured data, they are not stable as well. However, to detect the problems that occur in the system by checking one log by one log, it's complicated and boring work for the system operators. Using machine learning techniques and natural language processing techniques to detect the HDFS block anomaly will help the system operators to locate and fix the anomaly rapidly and accurately. This paper proposes a streaming HDFS log block anomaly workflow. It helps maintenance practitioners to use parallel computing network in processing historical log, and construct LLM-BiLSTM hybrid deep learning model to detect anomaly block in HDFS, then build streaming log pipeline based on Kafka to give one real-time HDFS log block anomaly detection solution.