# VirnyFlow: Optimizing ML Pipelines for Accuracy, Fairness, and Stability at Scale

arXiv: https://arxiv.org/abs/2506.01584 · HTML: https://mlsanity.com/paper/2506.01584/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 91 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Builds on (most cited first, 30 of 36)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle](https://mlsanity.com/paper/1901.10002/) (2019)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [FLAML: A Fast and Lightweight AutoML Library](https://mlsanity.com/paper/1911.04706/) (2019)
- [A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizations](https://mlsanity.com/paper/1805.12168/) (2018)
- [Problem Formulation and Fairness](https://mlsanity.com/paper/1901.02547/) (2019)
- [Max-value Entropy Search for Multi-Objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/2009.01721/) (2020)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [Interactive Model Cards: A Human-Centered Approach to Model Documentation](https://mlsanity.com/paper/2205.02894/) (2022)
- [Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization](https://mlsanity.com/paper/2006.04655/) (2020)
- [Multi-objective Bayesian Optimization using Pareto-frontier Entropy](https://mlsanity.com/paper/1906.00127/) (2019)
- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [OpenBox: A Generalized Black-box Optimization Service](https://mlsanity.com/paper/2106.00421/) (2021)
- [Fair Bayesian Optimization](https://mlsanity.com/paper/2006.05109/) (2020)
- [Uncertainty-Aware Search Framework for Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2204.05944/) (2022)
- [Multi-Objective Meta Learning](https://mlsanity.com/paper/2102.07121/) (2021)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [Fairness through Aleatoric Uncertainty](https://mlsanity.com/paper/2304.03646/) (2023)
- [Fair and Green Hyperparameter Optimization via Multi-objective and Multiple Information Source Bayesian Optimization](https://mlsanity.com/paper/2205.08835/) (2022)

## Abstract (arXiv)

Developing machine learning (ML) systems for real-world deployment requires navigating context-dependent trade-offs among accuracy, fairness, stability, and other objectives. Existing AutoML frameworks optimize pipelines efficiently, but they fix the optimization objective up front, leave it outside the developer's control during search, and rarely scale beyond a single node. We present VirnyFlow, a system that optimizes ML pipelines jointly for accuracy, fairness, and stability at scale. A user-defined evaluation protocol, with fairness measured over binary and intersectional groups, drives every layer of the optimizer: multi-objective Bayesian optimization of physical pipelines, cost-aware bandit selection of logical pipelines, and multi-criterion pruning. The architecture combines asynchronous execution over Apache Kafka with database-backed experiment management, providing fine-grained parallelism, fault tolerance, and interactive inspection of trade-offs. On six real-world datasets, VirnyFlow achieves competitive or superior performance compared to state-of-the-art AutoML systems (auto-sklearn, Alpine Meadow, FLAML) under identical resource constraints, scales to 128 workers across four nodes on datasets of up to 2.6M records, and achieves up to 7x higher speedup than the best-scaling single-node baseline, while maintaining stable accuracy and fairness as parallelism increases. A clinical case study on distribution shift and an IRB-approved user study demonstrate human-in-the-loop navigation of trade-offs in practice: rather than returning a single "best" model, VirnyFlow lets data scientists define, inspect, and iteratively refine the objectives of the search to fit their deployment context.