mlsanityData to 7 Oct 2026

On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models

arXiv 2209.05310 ↗First posted 12 Sep 2022cs.IR cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

For industrial-scale advertising systems, prediction of ad click-through rate (CTR) is a central problem. Ad clicks constitute a significant class of user engagements and are often used as the primary signal for the usefulness of ads to users. Additionally, in cost-per-click advertising systems where advertisers are charged per click, click rate expectations feed directly into value estimation. Accordingly, CTR model development is a significant investment for most Internet advertising companies. Engineering for such problems requires many machine learning (ML) techniques suited to online learning that go well beyond traditional accuracy improvements, especially concerning efficiency, reproducibility, calibration, credit attribution. We present a case study of practical techniques deployed in Google's search ads CTR model. This paper provides an industry case study highlighting important areas of current ML research and illustrating how impactful new ML methods are evaluated and made useful in a large-scale industrial setting.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 44 of its 75 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

44 of 75 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  6. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  7. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  10. 2017Deep Interest Network for Click-Through Rate Predictioncited by 541
  11. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  12. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
32 more
  1. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  2. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 326
  3. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  4. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  5. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  6. 2013Predicting Parameters in Deep Learningcited by 223
  7. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  8. 2016Unbiased Learning-to-Rank with Biased Feedbackcited by 155
  9. 2018Large scale distributed neural network training through online distillationcited by 144
  10. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  11. 2017Continuously Differentiable Exponential Linear Unitscited by 73
  12. 2020Can weight sharing outperform random architecture search? An investigation with TuNAScited by 68
  13. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  14. 2021Nondeterminism and Instability in Neural Network Optimizationcited by 47
  15. 2019Analyzing the Role of Model Uncertainty for Electronic Health Recordscited by 41
  16. 2016Scalable Learning of Non-Decomposable Objectivescited by 40
  17. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 39
  18. 2020Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Modelscited by 39
  19. 2018TF-Ranking: Scalable TensorFlow Library for Learning-to-Rankcited by 32
  20. 2020Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Trainingcited by 30
  21. 2021On the Reproducibility of Neural Network Predictionscited by 28
  22. 2020On Power Laws in Deep Ensemblescited by 26
  23. 2022Reproducibility in Optimization: Theoretical Framework and Limitscited by 21
  24. 2020When Ensembling Smaller Models is More Efficient than Single Large Modelscited by 19
  25. 2020Scalable Second Order Optimization for Deep Learningcited by 16
  26. 2020Smooth activations and reproducibility in deep networkscited by 15
  27. 2013Local case-control sampling: Efficient subsampling in imbalanced data setscited by 13
  28. 2020Anti-Distillation: Improving reproducibility of deep networkscited by 13
  29. 2021Synthesizing Irreproducibility in Deep Networkscited by 9
  30. 2016Large-scale Validation of Counterfactual Learning Methods: A Test-Bedcited by 8
  31. 2020Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systemscited by 8
  32. 2017Sponsored Search Auctions with Rich Adscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 17; 2024: 7; 2025: 6; 2026: 3.

34 papers.

Sort:

  1. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  2. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  3. 2024A New Perspective on Shampoo's Preconditionercited by 54
  4. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 29
  5. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 16
  6. 2023Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systemscited by 14
  7. 2023MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systemscited by 14
  8. 2025External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendationcited by 13
  9. 2023Improving Training Stability for Multitask Ranking Models in Recommender Systemscited by 12
  10. 2024LiRank: Industrial Large Scale Ranking Models at LinkedIncited by 12
  11. 2023Rec4Ad: A Free Lunch to Mitigate Sample Selection Bias for Ads CTR Prediction in Taobaocited by 11
  12. 2022RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructurecited by 9
22 more
  1. 2023MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendationcited by 9
  2. 2023Learning Rate Schedules in the Presence of Distribution Shiftcited by 7
  3. 2023What do larger image classifiers memorise?cited by 6
  4. 2023Reasonable Scale Machine Learning with Open-Source Metaflowcited by 5
  5. 2023FlexShard: Flexible Sharding for Industry-Scale Sequence Recommendation Modelscited by 4
  6. 2023Rankitect: Ranking Architecture Search Battling World-class Engineers at Meta Scalecited by 3
  7. 2023TwERC: High Performance Ensembled Candidate Generation for Ads Recommendation at Twittercited by 2
  8. 2023KrADagrad: Kronecker Approximation-Domination Gradient Preconditioned Stochastic Optimizationcited by 2
  9. 2024PriorBoost: An Adaptive Algorithm for Learning from Aggregate Responsescited by 2
  10. 2024Data Efficiency for Large Recommendation Modelscited by 2
  11. 2025Matryoshka Model Learning for Improved Elastic Student Modelscited by 2
  12. 2025Taming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-trainingcited by 2
  13. 2025Powering Job Search at Scale: LLM-Enhanced Query Understanding in Job Matching Systemscited by 2
  14. 2023Pairwise Ranking Losses of Click-Through Rates Prediction for Welfare Maximization in Ad Auctionscited by 1
  15. 2025iLTM: Integrated Large Tabular Modelcited by 1
  16. 2023Unleash the Power of Context: Enhancing Large-Scale Recommender Systems with Context-Based Prediction Modelscited by 0
  17. 2024Unified Low-rank Compression Framework for Click-through Rate Predictioncited by 0
  18. 2024CubicML: Automated ML for Large ML Systems Co-design with ML Prediction of Performancecited by 0
  19. 2025Beyond Self-Consistency: Loss-Balanced Perturbation-Based Regularization Improves Industrial-Scale Ads Rankingcited by 0
  20. 2026CADET: Context-Conditioned Ads CTR Prediction With a Decoder-Only Transformercited by 0
  21. 2026Multi-Probe Zero Collision Hash (MPZCH): Mitigating Embedding Collisions and Enhancing Model Freshness in Large-Scale Recommenderscited by 0
  22. 2026LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendationcited by 0