# Hercules: Heterogeneity-Aware Inference Serving for At-Scale Personalized Recommendation

arXiv: https://arxiv.org/abs/2203.07424 · HTML: https://mlsanity.com/paper/2203.07424/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2022.

## Built on by (most cited first, 9 of 9)

- [MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation](https://mlsanity.com/paper/2302.10872/) (2023)
- [PreSto: An In-Storage Data Preprocessing System for Training Recommendation Models](https://mlsanity.com/paper/2406.14571/) (2024)
- [xGR: Efficient Generative Recommendation Serving at Scale](https://mlsanity.com/paper/2512.11529/) (2025)
- [PIFS-Rec: Process-In-Fabric-Switch for Large-Scale Recommendation System Inferences](https://mlsanity.com/paper/2409.16633/) (2024)
- [ProactivePIM: Accelerating Weight-Sharing Embedding Layer with PIM for Scalable Recommendation System](https://mlsanity.com/paper/2402.04032/) (2024)
- [ElasticRec: A Microservice-based Model Serving Architecture Enabling Elastic Resource Scaling for Recommendation Models](https://mlsanity.com/paper/2406.06955/) (2024)
- [Pushing the Performance Envelope of DNN-based Recommendation Systems Inference on GPUs](https://mlsanity.com/paper/2410.22249/) (2024)
- [TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading](https://mlsanity.com/paper/2603.01058/) (2026)
- [ZoRRO: A Zero-Weight Personalized Recommender System for Scalable News Recommendation](https://mlsanity.com/paper/2607.10910/) (2026)

## Abstract (arXiv)

Personalized recommendation is an important class of deep-learning applications that powers a large collection of internet services and consumes a considerable amount of datacenter resources. As the scale of production-grade recommendation systems continues to grow, optimizing their serving performance and efficiency in a heterogeneous datacenter is important and can translate into infrastructure capacity saving. In this paper, we propose Hercules, an optimized framework for personalized recommendation inference serving that targets diverse industry-representative models and cloud-scale heterogeneous systems. Hercules performs a two-stage optimization procedure - offline profiling and online serving. The first stage searches the large under-explored task scheduling space with a gradient-based search algorithm achieving up to 9.0x latency-bounded throughput improvement on individual servers; it also identifies the optimal heterogeneous server architecture for each recommendation workload. The second stage performs heterogeneity-aware cluster provisioning to optimize resource mapping and allocation in response to fluctuating diurnal loads. The proposed cluster scheduler in Hercules achieves 47.7% cluster capacity saving and reduces the provisioned power by 23.7% over a state-of-the-art greedy scheduler.