# OpenMLDB: A Real-Time Relational Data Feature Computation System for Online ML

arXiv: https://arxiv.org/abs/2501.08591 · HTML: https://mlsanity.com/paper/2501.08591/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 43 references that are in the map.
- First posted on arXiv 15 Jan 2025.

## Builds on (most cited first, 5 of 5)

- [An End-to-End Learning-based Cost Estimator](https://mlsanity.com/paper/1906.02560/) (2019)
- [Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources](https://mlsanity.com/paper/1802.10233/) (2018)
- [Using LLVM-based JIT Compilation in Genetic Programming](https://mlsanity.com/paper/1701.05730/) (2017)
- [Compiling a Higher-Order Smart Contract Language to LLVM](https://mlsanity.com/paper/2008.05555/) (2020)
- [An Empirical Analysis of Just-in-Time Compilation in Modern Databases](https://mlsanity.com/paper/2311.04692/) (2023)

## Built on by (most cited first, 3 of 3)

- [DBAIOps: A Reasoning LLM-Enhanced Database Operation and Maintenance System using Knowledge Graphs](https://mlsanity.com/paper/2508.01136/) (2025)
- [Automating Database-Native Function Code Synthesis with LLMs](https://mlsanity.com/paper/2604.06231/) (2026)
- [FeatInsight: An Online ML Feature Management System on 4Paradigm Sage-Studio Platform](https://mlsanity.com/paper/2504.00786/) (2025)

## Abstract (arXiv)

Efficient and consistent feature computation is crucial for a wide range of online ML applications. Typically, feature computation is divided into two distinct phases, i.e., offline stage for model training and online stage for model serving. These phases often rely on execution engines with different interface languages and function implementations, causing significant inconsistencies. Moreover, many online ML features involve complex time-series computations (e.g., functions over varied-length table windows) that differ from standard streaming and analytical queries. Existing data processing systems (e.g., Spark, Flink, DuckDB) often incur multi-second latencies for these computations, making them unsuitable for real-time online ML applications that demand timely feature updates. This paper presents OpenMLDB, a feature computation system deployed in 4Paradigm's SageOne platform and over 100 real scenarios. Technically, OpenMLDB first employs a unified query plan generator for consistent computation results across the offline and online stages, significantly reducing feature deployment overhead. Second, OpenMLDB provides an online execution engine that resolves performance bottlenecks caused by long window computations (via pre-aggregation) and multi-table window unions (via data self-adjusting). It also provides a high-performance offline execution engine with window parallel optimization and time-aware data skew resolving. Third, OpenMLDB features a compact data format and stream-focused indexing to maximize memory usage and accelerate data access. Evaluations in testing and real workloads reveal significant performance improvements and resource savings compared to the baseline systems. The open community of OpenMLDB now has over 150 contributors and gained 1.6k stars on GitHub.