mlsanityData to 7 Oct 2026

Improving Inference Performance of Machine Learning with the Divide-and-Conquer Principle

arXiv 2301.05099 ↗First posted 12 Jan 2023cs.LG cs.AI cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Many popular machine learning models scale poorly when deployed on CPUs. In this paper we explore the reasons why and propose a simple, yet effective approach based on the well-known Divide-and-Conquer Principle to tackle this problem of great practical importance. Given an inference job, instead of using all available computing resources (i.e., CPU cores) for running it, the idea is to break the job into independent parts that can be executed in parallel, each with the number of cores according to its expected computational cost. We implement this idea in the popular OnnxRuntime framework and evaluate its effectiveness with several use cases, including the well-known models for optical character recognition (PaddleOCR) and natural language processing (BERT).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 12 Jan 2023.

Builds on

15 of 30 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2016Layer Normalizationcited by 5,579
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  7. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  8. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 147
  9. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  10. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  11. 2021Efficient Multi-Stage Video Denoising with Recurrent Spatio-Temporal Fusioncited by 36
  12. 2018Optimizing CNN Model Inference on CPUscited by 29
3 more
  1. 2019Exploiting Parallelism Opportunities with Deep Learning Frameworkscited by 10
  2. 2021Optimizing Inference Performance of Transformers on CPUscited by 7
  3. 2015Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Surveycited by 6

Built on by

Not cited yet.