# Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference

arXiv: https://arxiv.org/abs/2210.08803 · HTML: https://mlsanity.com/paper/2210.08803/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference](https://mlsanity.com/paper/2001.02772/) (2020)
- [Distributed Training of Deep Learning Models: A Taxonomic Perspective](https://mlsanity.com/paper/2007.03970/) (2020)
- [Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training](https://mlsanity.com/paper/1805.07891/) (2018)
- [Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism](https://mlsanity.com/paper/2010.08899/) (2020)
- [ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table](https://mlsanity.com/paper/2104.08542/) (2021)

## Built on by (most cited first, 20 of 20)

- [Dynamic Sparse Training with Structured Sparsity](https://mlsanity.com/paper/2305.02299/) (2023)
- [RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference](https://mlsanity.com/paper/2505.02922/) (2025)
- [Experimental Analysis of Large-scale Learnable Vector Storage Compression](https://mlsanity.com/paper/2311.15578/) (2023)
- [CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models](https://mlsanity.com/paper/2312.03256/) (2023)
- [Sparsity-Preserving Differentially Private Training of Large Embedding Models](https://mlsanity.com/paper/2311.08357/) (2023)
- [G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale Recommender Systems](https://mlsanity.com/paper/2401.04338/) (2024)
- [InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models](https://mlsanity.com/paper/2308.08500/) (2023)
- [Toward Robust and Efficient ML-Based GPU Caching for Modern Inference](https://mlsanity.com/paper/2509.20979/) (2025)
- [Saturn: An Optimized Data System for Large Model Deep Learning Workloads](https://mlsanity.com/paper/2309.01226/) (2023)
- [Differentially Private Optimization with Sparse Gradients](https://mlsanity.com/paper/2404.10881/) (2024)
- [Research on Low-Latency Inference and Training Efficiency Optimization for Graph Neural Network and Large Language Model-Based Recommendation Systems](https://mlsanity.com/paper/2507.01035/) (2025)
- [RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models](https://mlsanity.com/paper/2509.20883/) (2025)
- [Failure Tolerant Training with Persistent Memory Disaggregation over CXL](https://mlsanity.com/paper/2301.07492/) (2023)
- [Fine-Grained Embedding Dimension Optimization During Training for Recommender Systems](https://mlsanity.com/paper/2401.04408/) (2024)
- [PIFS-Rec: Process-In-Fabric-Switch for Large-Scale Recommendation System Inferences](https://mlsanity.com/paper/2409.16633/) (2024)
- [ElasticRec: A Microservice-based Model Serving Architecture Enabling Elastic Resource Scaling for Recommendation Models](https://mlsanity.com/paper/2406.06955/) (2024)
- [SCRec: A Scalable Computational Storage System with Statistical Sharding and Tensor-train Decomposition for Recommendation Models](https://mlsanity.com/paper/2504.00520/) (2025)
- [MLKV: Efficiently Scaling up Large Embedding Model Training with Disk-based Key-Value Storage](https://mlsanity.com/paper/2504.01506/) (2025)
- [Faster Distributed Inference-Only Recommender Systems via Bounded Lag Synchronous Collectives](https://mlsanity.com/paper/2512.19342/) (2025)
- [NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining](https://mlsanity.com/paper/2604.06956/) (2026)

## Abstract (arXiv)

In this talk, we introduce Merlin HugeCTR. Merlin HugeCTR is an open source, GPU-accelerated integration framework for click-through rate estimation. It optimizes both training and inference, whilst enabling model training at scale with model-parallel embeddings and data-parallel neural networks. In particular, Merlin HugeCTR combines a high-performance GPU embedding cache with an hierarchical storage architecture, to realize low-latency retrieval of embeddings for online model inference tasks. In the MLPerf v1.0 DLRM model training benchmark, Merlin HugeCTR achieves a speedup of up to 24.6x on a single DGX A100 (8x A100) over PyTorch on 4x4-socket CPU nodes (4x4x28 cores). Merlin HugeCTR can also take advantage of multi-node environments to accelerate training even further. Since late 2021, Merlin HugeCTR additionally features a hierarchical parameter server (HPS) and supports deployment via the NVIDIA Triton server framework, to leverage the computational capabilities of GPUs for high-speed recommendation model inference. Using this HPS, Merlin HugeCTR users can achieve a 5~62x speedup (batch size dependent) for popular recommendation models over CPU baseline implementations, and dramatically reduce their end-to-end inference latency.