RPU -- A Reasoning Processing Unit
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Large language model (LLM) inference performance is increasingly bottlenecked by the memory wall. While GPUs continue to scale raw compute throughput, they struggle to deliver scalable performance for memory bandwidth bound workloads. This challenge is amplified by emerging reasoning LLM applications, where long output sequences, low arithmetic intensity, and tight latency constraints demand significantly higher memory bandwidth. As a result, system utilization drops and energy per inference rises, highlighting the need for an optimized system architecture for scalable memory bandwidth. To address these challenges we present the Reasoning Processing Unit (RPU), a chiplet-based architecture designed to address the challenges of the modern memory wall. RPU introduces: (1) A Capacity-Optimized High-Bandwidth Memory (HBM-CO) that trades capacity for lower energy and cost; (2) a scalable chiplet architecture featuring a bandwidth-first power and area provisioning design; and (3) a decoupled microarchitecture that separates memory, compute, and communication pipelines to sustain high bandwidth utilization. Simulation results show that RPU performs up to 45.3x lower latency and 18.6x higher throughput over an H100 system at ISO-TDP on Llama3-405B.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 21 of its 71 references that are in the map. First posted on arXiv 20 Feb 2026.
Builds on
21 of 71 references.
Closest ancestors · 3
Same direction, newest first
- 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longer21 Feb
- 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs30 Jan
- 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
Other references · 18
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
- 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
- 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
- 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 69
- 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
- 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 48
- 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
- 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
- 2025Mitigating Response Delays in Free-Form Conversations with LLM-powered Intelligent Virtual Agentscited by 12
6 more
- 2024Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inferencecited by 9
- 2024Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Modelscited by 5
- 2023CAMEL: Co-Designing AI Models and Embedded DRAMs for Efficient On-Device Learningcited by 3
- 2018Exploiting the DRAM Microarchitecture to Increase Memory-Level Parallelismcited by 1
- 2025Good things come in small packages: Should we build AI clusters with Lite-GPUs?cited by 1
- 2025A Performance Model for Warp Specialization Kernelscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles
- 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
- 2025Reasoning Language Model Inference Serving Unveiled: An Empirical Study
- 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
- 2026CHIPSMORE: Compute-in-Interconnect and -Memory Chiplets for Multi-Mode Multi-Request LLM Inference Acceleration
- 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
- 2025EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs
- 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling