# ESTemd: A Distributed Processing Framework for Environmental Monitoring based on Apache Kafka Streaming Engine

arXiv: https://arxiv.org/abs/2104.01082 · HTML: https://mlsanity.com/paper/2104.01082/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 26 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 6 of 6)

- [Beyond Batch Processing: Towards Real-Time and Streaming Big Data](https://mlsanity.com/paper/1403.3375/) (2014)
- [Towards Semantic Integration of Heterogeneous Sensor Data with Indigenous Knowledge for Drought Forecasting](https://mlsanity.com/paper/1601.01920/) (2016)
- [Real Time Analytics: Algorithms and Systems](https://mlsanity.com/paper/1708.02621/) (2017)
- [Semantic Interoperability Middleware Architecture for Heterogeneous Environmental Data Sources](https://mlsanity.com/paper/1809.05890/) (2018)
- [Benchmarking Distributed Stream Processing Platforms for IoT Applications](https://mlsanity.com/paper/1606.07621/) (2016)
- [Knowledge-infused and Consistent Complex Event Processing over Real-time and Persistent Streams](https://mlsanity.com/paper/1611.00676/) (2016)

## Built on by (most cited first, 1 of 1)

- [A Blockchain-based Model for Securing Data Pipeline in a Heterogeneous Information System](https://mlsanity.com/paper/2401.09240/) (2024)

## Abstract (arXiv)

Distributed networks and real-time systems are becoming the most important components for the new computer age, the Internet of Things (IoT), with huge data streams or data sets generated from sensors and data generated from existing legacy systems. The data generated offers the ability to measure, infer and understand environmental indicators, from delicate ecologies and natural resources to urban environments. This can be achieved through the analysis of the heterogeneous data sources (structured and unstructured). In this paper, we propose a distributed framework Event STream Processing Engine for Environmental Monitoring Domain (ESTemd) for the application of stream processing on heterogeneous environmental data. Our work in this area demonstrates the useful role big data techniques can play in an environmental decision support system, early warning and forecasting systems. The proposed framework addresses the challenges of data heterogeneity from heterogeneous systems and real time processing of huge environmental datasets through a publish/subscribe method via a unified data pipeline with the application of Apache Kafka for real time analytics.