# JoinBoost: Grow Trees Over Normalized Data Using Only SQL

arXiv: https://arxiv.org/abs/2307.00422 · HTML: https://mlsanity.com/paper/2307.00422/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 81 references that are in the map.
- First posted on arXiv 1 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [Operationalizing Machine Learning: An Interview Study](https://mlsanity.com/paper/2209.09125/) (2022)
- [FAQ: Questions Asked Frequently](https://mlsanity.com/paper/1504.04044/) (2015)
- [Towards Linear Algebra over Normalized Data](https://mlsanity.com/paper/1612.07448/) (2016)
- [PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference](https://mlsanity.com/paper/1709.09216/) (2017)
- [Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning](https://mlsanity.com/paper/1904.11121/) (2019)
- [A Layered Aggregate Engine for Analytics Workloads](https://mlsanity.com/paper/1906.08687/) (2019)
- [Benchmarking and Optimization of Gradient Boosting Decision Tree Algorithms](https://mlsanity.com/paper/1809.04559/) (2018)
- [Share the Tensor Tea: How Databases can Leverage the Machine Learning Ecosystem](https://mlsanity.com/paper/2209.04579/) (2022)
- [AC/DC: In-Database Learning Thunderstruck](https://mlsanity.com/paper/1803.07480/) (2018)
- [Rk-means: Fast Clustering for Relational Data](https://mlsanity.com/paper/1910.04939/) (2019)
- [Causal Gradient Boosting: Boosted Instrumental Variable Regression](https://mlsanity.com/paper/2101.06078/) (2021)
- [TCUDB: Accelerating Database with Tensor Processors](https://mlsanity.com/paper/2112.07552/) (2021)
- [Functional Aggregate Queries with Additive Inequalities](https://mlsanity.com/paper/1812.09526/) (2018)
- [Aggregations over Generalized Hypertree Decompositions](https://mlsanity.com/paper/1508.07532/) (2015)
- [Calibration: A Simple Trick for Wide-table Delta Analytics](https://mlsanity.com/paper/2210.03851/) (2022)

## Built on by (most cited first, 4 of 4)

- [In-Database Data Imputation](https://mlsanity.com/paper/2401.03359/) (2024)
- [Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines](https://mlsanity.com/paper/2405.11191/) (2024)
- [InferF: Declarative Factorization of AI/ML Inferences over Joins](https://mlsanity.com/paper/2511.20489/) (2025)
- [Fast Factorized Learning: Powered by In-Memory Database Systems](https://mlsanity.com/paper/2512.09836/) (2025)

## Abstract (arXiv)

Although dominant for tabular data, ML libraries that train tree models over normalized databases (e.g., LightGBM, XGBoost) require the data to be denormalized as a single table, materialized, and exported. This process is not scalable, slow, and poses security risks. In-DB ML aims to train models within DBMSes to avoid data movement and provide data governance. Rather than modify a DBMS to support In-DB ML, is it possible to offer competitive tree training performance to specialized ML libraries...with only SQL? We present JoinBoost, a Python library that rewrites tree training algorithms over normalized databases into pure SQL. It is portable to any DBMS, offers performance competitive with specialized ML libraries, and scales with the underlying DBMS capabilities. JoinBoost extends prior work from both algorithmic and systems perspectives. Algorithmically, we support factorized gradient boosting, by updating the Y variable to the residual in the non-materialized join result. Although this view update problem is generally ambiguous, we identify addition-to-multiplication preserving, the key property of variance semi-ring to support rmse, the most widely used criterion. System-wise, we identify residual updates as a performance bottleneck. Such overhead can be natively minimized on columnar DBMSes by creating a new column of residual values and adding it as a projection. We validate this with two implementations on DuckDB, with no or minimal modifications to its internals for portability. Our experiment shows that JoinBoost is 3x (1.1x) faster for random forests (gradient boosting) compared to LightGBM, and over an order magnitude faster than state-of-the-art In-DB ML systems. Further, JoinBoost scales well beyond LightGBM in terms of the # features, DB size (TPC-DS SF=1000), and join graph complexity (galaxy schemas).