# AutoStreamPipe: LLM Assisted Automatic Generation of Data Stream Processing Pipelines

arXiv: https://arxiv.org/abs/2510.23408 · HTML: https://mlsanity.com/paper/2510.23408/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 70 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [A Survey on Code Generation with LLM-based Agents](https://mlsanity.com/paper/2508.00083/) (2025)
- [Answering Questions by Meta-Reasoning over Multiple Chains of Thought](https://mlsanity.com/paper/2304.13007/) (2023)
- [AutoFlow: Automated Workflow Generation for Large Language Model Agents](https://mlsanity.com/paper/2407.12821/) (2024)
- [LLMs for Science: Usage for Code Generation and Data Analysis](https://mlsanity.com/paper/2311.16733/) (2023)
- [Data Stream Clustering: A Review](https://mlsanity.com/paper/2007.10781/) (2020)
- [BigDataBench: a Big Data Benchmark Suite from Internet Services](https://mlsanity.com/paper/1401.1406/) (2014)
- [A Multi-Stage Automated Online Network Data Stream Analytics Framework for IIoT Systems](https://mlsanity.com/paper/2210.01985/) (2022)
- [Layer-of-Thoughts Prompting (LoT): Leveraging LLM-Based Retrieval with Constraint Hierarchies](https://mlsanity.com/paper/2410.12153/) (2024)
- [A Survey on the Evolution of Stream Processing Systems](https://mlsanity.com/paper/2008.00842/) (2020)
- [PipeGen: Data Pipe Generator for Hybrid Analytics](https://mlsanity.com/paper/1605.01664/) (2016)
- [RIoTBench: A Real-time IoT Benchmark for Distributed Stream Processing Platforms](https://mlsanity.com/paper/1701.08530/) (2017)
- [LLM4FaaS: No-Code Application Development using LLMs and FaaS](https://mlsanity.com/paper/2502.14450/) (2025)
- [WfChef: Automated Generation of Accurate Scientific Workflow Generators](https://mlsanity.com/paper/2105.00129/) (2021)
- [PDSP-Bench: A Benchmarking System for Parallel and Distributed Stream Processing](https://mlsanity.com/paper/2504.10704/) (2025)

## Built on by (most cited first, 2 of 2)

- [Can Large Language Models Reason about Event-Time Stream-Processing Semantics?](https://mlsanity.com/paper/2608.12348/) (2026)
- [ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time](https://mlsanity.com/paper/2610.02363/) (2026)

## Abstract (arXiv)

Data pipelines are essential in stream processing as they enable the efficient collection, processing, and delivery of real-time data, supporting rapid data analysis. In this paper, we present AutoStreamPipe, a novel framework that employs Large Language Models (LLMs) to automate the design, generation, and deployment of stream processing pipelines. AutoStreamPipe bridges the semantic gap between high-level user intent and platform-specific implementations across distributed stream processing systems for structured multi-agent reasoning by integrating a Hypergraph of Thoughts (HGoT) as an extended version of GoT. AutoStreamPipe combines resilient execution strategies, advanced query analysis, and HGoT to deliver pipelines with good accuracy. Experimental evaluations on diverse pipelines demonstrate that AutoStreamPipe significantly reduces development time (x6.3) and error rates (x5.19), as measured by a novel Error-Free Score (EFS), compared to LLM code-generation methods.