Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT
Not on the map. Why?
Abstract (arXiv)
Scientific workflow systems are increasingly popular for expressing and executing complex data analysis pipelines over large datasets, as they offer reproducibility, dependability, and scalability of analyses by automatic parallelization on large compute clusters. However, implementing workflows is difficult due to the involvement of many black-box tools and the deep infrastructure stack necessary for their execution. Simultaneously, user-supporting tools are rare, and the number of available examples is much lower than in classical programming languages. To address these challenges, we investigate the efficiency of Large Language Models (LLMs), specifically ChatGPT, to support users when dealing with scientific workflows. We performed three user studies in two scientific domains to evaluate ChatGPT for comprehending, adapting, and extending workflows. Our results indicate that LLMs efficiently interpret workflows but achieve lower performance for exchanging components or purposeful workflow extensions. We characterize their limitations in these challenging scenarios and suggest future research directions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 40 of its 113 references that are in the map. First posted on arXiv 3 Nov 2023.
Builds on
40 of 113 references.
References · 40
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
- 2022Competition-Level Code Generation with AlphaCodecited by 1,462
- 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
- 2023PaLM 2 Technical Reportcited by 1,162
- 2023Teaching Large Language Models to Self-Debugcited by 988
28 more
- 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
- 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
- 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
- 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
- 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
- 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
- 2023An Analysis of the Automatic Bug Fixing Performance of ChatGPTcited by 110
- 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
- 2020Language Models are Open Knowledge Graphscited by 95
- 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 89
- 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
- 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
- 2023ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Designcited by 66
- 2023Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Modelscited by 59
- 2023"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Modelscited by 48
- 2021PopBlends: Strategies for Conceptual Blending with Large Language Modelscited by 22
- 2023ChatGPT as your Personal Data Scientistcited by 22
- 2023Human-in-the-Loop through Chain-of-Thoughtcited by 20
- 2023Human-in-the-Loop Schema Inductioncited by 13
- 2023"AI enhances our performance, I have no doubt this one will do the same": The Placebo effect is robust to negative descriptions of AIcited by 11
- 2022The Placebo Effect of Artificial Intelligence in Human-Computer Interactioncited by 9
- 2019VASTA: A Vision and Language-assisted Smartphone Task Automation Systemcited by 6
- 2019A Visual Programming Paradigm for Abstract Deep Learning Model Developmentcited by 5
- 2021A Community Roadmap for Scientific Workflows Research and Developmentcited by 5
- 2021Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Languagecited by 4
- 2023Correlates of Programmer Efficacy and Their Link to Experience: A Combined EEG and Eye-Tracking Studycited by 2
- 2023Laminar: A New Serverless Stream-based Framework with Semantic Code Search and Code Completioncited by 2
Built on by
Citing papers per year: 2025: 2.
2 papers.