mlsanityData to 14 Oct 2026

StreamLink: Large-Language-Model Driven Distributed Data Engineering System

arXiv 2505.21575 ↗First posted 27 May 2025cs.DB cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable proficiency in natural language understanding (NLU), opening doors for innovative applications. We introduce StreamLink - an LLM-driven distributed data system designed to improve the efficiency and accessibility of data engineering tasks. We build StreamLink on top of distributed frameworks such as Apache Spark and Hadoop to handle large data at scale. One of the important design philosophies of StreamLink is to respect user data privacy by utilizing local fine-tuned LLMs instead of a public AI service like ChatGPT. With help from domain-adapted LLMs, we can improve our system's understanding of natural language queries from users in various scenarios and simplify the procedure of generating database queries like the Structured Query Language (SQL) for information processing. We also incorporate LLM-based syntax and security checkers to guarantee the reliability and safety of each generated query. StreamLink illustrates the potential of merging generative LLMs with distributed data processing for comprehensive and user-centric data engineering. With this architecture, we allow users to interact with complex database systems at different scales in a user-friendly and security-ensured manner, where the SQL generation reaches over 10% of execution accuracy compared to baseline methods, and allow users to find the most concerned item from hundreds of millions of items within a few seconds using natural language.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

14 of 29 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  7. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  8. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  9. 2020Structure-Grounded Pretraining for Text-to-SQLcited by 126
  10. 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
  11. 2022RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQLcited by 85
  12. 2021Natural SQL: Making SQL Easier to Infer from Natural Language Specificationscited by 68
2 more
  1. 2022Importance of Synthesizing High-quality Data for Text-to-SQL Parsingcited by 28
  2. 2022N-Best Hypotheses Reranking for Text-To-SQL Systemscited by 19

Built on by

Not cited yet.