# A Layered Aggregate Engine for Analytics Workloads

arXiv: https://arxiv.org/abs/1906.08687 · HTML: https://mlsanity.com/paper/1906.08687/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2019.

## Built on by (most cited first, 12 of 12)

- [GeCo: Quality Counterfactual Explanations in Real Time](https://mlsanity.com/paper/2101.01292/) (2021)
- [Functional Collection Programming with Semi-Ring Dictionaries](https://mlsanity.com/paper/2103.06376/) (2021)
- [JoinBoost: Grow Trees Over Normalized Data Using Only SQL](https://mlsanity.com/paper/2307.00422/) (2023)
- [Approaching sales forecasting using recurrent neural networks and transformers](https://mlsanity.com/paper/2204.07786/) (2022)
- [In-Database Data Imputation](https://mlsanity.com/paper/2401.03359/) (2024)
- [The Duck's Brain: Training and Inference of Neural Networks in Modern Database Engines](https://mlsanity.com/paper/2312.17355/) (2023)
- [Fine-Tuning Data Structures for Analytical Query Processing](https://mlsanity.com/paper/2112.13099/) (2021)
- [Graphical Join: A New Physical Join Algorithm for RDBMSs](https://mlsanity.com/paper/2206.10435/) (2022)
- [TablePuppet: A Generic Framework for Relational Federated Learning](https://mlsanity.com/paper/2403.15839/) (2024)
- [InferF: Declarative Factorization of AI/ML Inferences over Joins](https://mlsanity.com/paper/2511.20489/) (2025)
- [Fast Factorized Learning: Powered by In-Memory Database Systems](https://mlsanity.com/paper/2512.09836/) (2025)
- [Data Flow Control: Data Safety Policies for AI Agents](https://mlsanity.com/paper/2606.05679/) (2026)

## Abstract (arXiv)

This paper introduces LMFAO (Layered Multiple Functional Aggregate Optimization), an in-memory optimization and execution engine for batches of aggregates over the input database. The primary motivation for this work stems from the observation that for a variety of analytics over databases, their data-intensive tasks can be decomposed into group-by aggregates over the join of the input database relations. We exemplify the versatility and competitiveness of LMFAO for a handful of widely used analytics: learning ridge linear regression, classification trees, regression trees, and the structure of Bayesian networks using Chow-Liu trees; and data cubes used for exploration in data warehousing. LMFAO consists of several layers of logical and code optimizations that systematically exploit sharing of computation, parallelism, and code specialization. We conducted two types of performance benchmarks. In experiments with four datasets, LMFAO outperforms by several orders of magnitude on one hand, a commercial database system and MonetDB for computing batches of aggregates, and on the other hand, TensorFlow, Scikit, R, and AC/DC for learning a variety of models over databases.