mlsanityData to 14 Oct 2026

How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF

arXiv 2607.19712 ↗First posted 22 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In RLHF pipelines, reward scoring blocks policy updates. Slow scoring bottlenecks the entire loop, since no update runs until every rollout gets a score. And yet most setups just default to PyTorch eager mode or torch.compile, no one checks if that's actually fastest. Scoring itself is small. Rollout generation eats far more of a typical RLHF step. But scoring and generation fight over the same CPU and GPU resources, so a faster scoring engine doesn't shrink step time on its own. It mainly frees up capacity generation can use instead. We built a native C++ inference engine on ONNX Runtime. First step: confirm correctness. Output matched the PyTorch reference to 5.7 x 10^-6 on CPU and 4.2 x 10^-3 on GPU, close enough to trust. Then we tested it against PyTorch eager mode, torch.compile, and FastAPI, on both CPU and GPU. CPU was decisive. Our engine beat every baseline, confidence intervals didn't even overlap. GPU gave a different view: we beat PyTorch and FastAPI, but torch.compile came out ahead. Further testing traced the speedup to ONNX Runtime itself, not C++ as a language. And batching strategy mattered more than either the language or the runtime choice, more than we expected. The results are from repeated, independent runs, since single runs just aren't reliable enough to trust.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 25 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

16 of 25 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  11. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
4 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  3. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
  4. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-turn RL with Structural and Performance Aware Rewards for CUDA Kernel Generation
  2. 2026KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?
  3. 2026Reinforcement Learning for Code Optimization
  4. 2026RLPF: Reinforcement Learning from Performance Feedback for Code Generation
  5. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
  6. 2025DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation
  7. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Models
  8. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains