# Parsl: Pervasive Parallel Programming in Python

arXiv: https://arxiv.org/abs/1905.02158 · HTML: https://mlsanity.com/paper/1905.02158/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2019.

## Built on by (most cited first, 30 of 36)

- [Empowering Scientific Workflows with Federated Agents](https://mlsanity.com/paper/2505.05428/) (2025)
- [HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights](https://mlsanity.com/paper/2505.04846/) (2025)
- [Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing](https://mlsanity.com/paper/2110.02827/) (2021)
- [Towards Lightweight Data Integration using Multi-workflow Provenance and Data Observability](https://mlsanity.com/paper/2308.09004/) (2023)
- [Pandemic Drugs at Pandemic Speed: Infrastructure for Accelerating COVID-19 Drug Discovery with Hybrid Machine Learning- and Physics-based Simulations on High Performance Computers](https://mlsanity.com/paper/2103.02843/) (2021)
- [The (R)evolution of Scientific Workflows in the Agentic AI Era: Towards Autonomous Science](https://mlsanity.com/paper/2509.09915/) (2025)
- [LSHBloom: Memory-efficient, Extreme-scale Document Deduplication](https://mlsanity.com/paper/2411.04257/) (2024)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)
- [Globus Automation Services: Research process automation across the space-time continuum](https://mlsanity.com/paper/2208.09513/) (2022)
- [AICCA: AI-driven Cloud Classification Atlas](https://mlsanity.com/paper/2209.15096/) (2022)
- [High Throughput Training of Deep Surrogates from Large Ensemble Runs](https://mlsanity.com/paper/2309.16743/) (2023)
- [Employing Artificial Intelligence to Steer Exascale Workflows with Colmena](https://mlsanity.com/paper/2408.14434/) (2024)
- [Flight: A FaaS-Based Framework for Complex and Hierarchical Federated Learning](https://mlsanity.com/paper/2409.16495/) (2024)
- [AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine](https://mlsanity.com/paper/2505.01435/) (2025)
- [Fantastic Scientific Agents and How to Build Them: AgentBuild for Rietveld Refinement](https://mlsanity.com/paper/2606.12834/) (2026)
- [Cloud Services Enable Efficient AI-Guided Simulation Workflows across Heterogeneous Resources](https://mlsanity.com/paper/2303.08803/) (2023)
- [Accelerating Electronic Stopping Power Predictions by 10 Million Times with a Combination of Time-Dependent Density Functional Theory and Machine Learning](https://mlsanity.com/paper/2311.00787/) (2023)
- [Multi-Agent Orchestration for High-Throughput Materials Screening on a Leadership-Class System](https://mlsanity.com/paper/2604.07681/) (2026)
- [AAFLOW: Scalable Patterns for Agentic AI Workflows](https://mlsanity.com/paper/2605.02162/) (2026)
- [[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows](https://mlsanity.com/paper/2607.10987/) (2026)
- [Reproducible and Portable Big Data Analytics in the Cloud](https://mlsanity.com/paper/2112.09762/) (2021)
- [Leveraging Large Language Models to Build and Execute Computational Workflows](https://mlsanity.com/paper/2312.07711/) (2023)
- [Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis](https://mlsanity.com/paper/2503.14356/) (2025)
- [ChemGraph-XANES: An Agentic Framework for XANES Simulation and Curation](https://mlsanity.com/paper/2604.16205/) (2026)
- [Automated Data Readiness for Scientific AI](https://mlsanity.com/paper/2607.02771/) (2026)
- [OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows](https://mlsanity.com/paper/2608.08340/) (2026)
- [Report of the Workshop on Program Synthesis for Scientific Computing](https://mlsanity.com/paper/2102.01687/) (2021)
- [Scalable Agentic Reasoning for Designing Biologics Targeting Intrinsically Disordered Proteins](https://mlsanity.com/paper/2512.15930/) (2025)
- [Twelve quick tips for designing AI-driven HPC workflows](https://mlsanity.com/paper/2606.07491/) (2026)
- [An Agentic AI Framework to Accelerate Scientific Discovery in Plant Phenotyping](https://mlsanity.com/paper/2606.31831/) (2026)

## Abstract (arXiv)

High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250 000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.