# From Large to Small: Transferring CUDA Optimization Expertise via Reasoning Graph

arXiv: https://arxiv.org/abs/2510.19873 · HTML: https://mlsanity.com/paper/2510.19873/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- Retrieval-augmented generation / [RAG for code](https://mlsanity.com/rag/rag-for-code/)

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning](https://mlsanity.com/paper/2405.00451/) (2024)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability](https://mlsanity.com/paper/2410.11414/) (2024)
- [On Protecting the Data Privacy of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2403.05156/) (2024)
- [Kevin: Multi-Turn RL for Generating CUDA Kernels](https://mlsanity.com/paper/2507.11948/) (2025)
- [RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation](https://mlsanity.com/paper/2409.09584/) (2024)
- [EVOR: Evolving Retrieval for Code Generation](https://mlsanity.com/paper/2402.12317/) (2024)
- [MCTS-RAG: Enhancing Retrieval-Augmented Generation with Monte Carlo Tree Search](https://mlsanity.com/paper/2503.20757/) (2025)
- [Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions](https://mlsanity.com/paper/2408.05212/) (2024)
- [A Comparative Analysis of Fine-Tuned LLMs and Few-Shot Learning of LLMs for Financial Sentiment Analysis](https://mlsanity.com/paper/2312.08725/) (2023)
- [Deploying Open-Source Large Language Models: A performance Analysis](https://mlsanity.com/paper/2409.14887/) (2024)
- [Leveraging Reasoning Model Answers to Enhance Non-Reasoning Model Capability](https://mlsanity.com/paper/2504.09639/) (2025)

## Built on by (most cited first, 10 of 10)

- [CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation](https://mlsanity.com/paper/2602.24286/) (2026)
- [KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels](https://mlsanity.com/paper/2605.04956/) (2026)
- [Towards Automated Kernel Generation in the Era of LLMs](https://mlsanity.com/paper/2601.15727/) (2026)
- [HTAM: Hierarchical Transition-Attended Memory for Operator Optimization](https://mlsanity.com/paper/2605.29734/) (2026)
- [From Tokens to Regions: CUDA-Sensitive Instruction Tuning for GPU Kernel Generation](https://mlsanity.com/paper/2606.16231/) (2026)
- [VeriPilot: An LLM-Powered Verilog Debugging Framework](https://mlsanity.com/paper/2606.23759/) (2026)
- [Step-TP: A Grounded, Step-Level Dataset with Chain-of-Thought Reasoning for LLM-Guided Tensor Program Optimization](https://mlsanity.com/paper/2605.25954/) (2026)
- [CUDA-Harness: Harnessing Agentic CUDA Kernel Generation and Optimization from Natural Language](https://mlsanity.com/paper/2609.00058/) (2026)
- [HyperMCTS: Hypergraph-Augmented MCTS for Long-Horizon LLM Agents](https://mlsanity.com/paper/2609.33920/) (2026)
- [MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code](https://mlsanity.com/paper/2610.05014/) (2026)

## Abstract (arXiv)

Despite significant evolution of CUDA programming and domain-specific libraries, effectively utilizing GPUs with massively parallel engines remains difficult. Large language models (LLMs) show strong potential in generating optimized CUDA code from sequential code. However, using LLMs in practice faces two major challenges: cloud-based APIs pose risks of code leakage, and local deployment is often computationally expensive and inefficient. These drawbacks have spurred interest in small language models (SLMs), which are more lightweight and privacy-friendly. Encouragingly, recent studies show that SLMs can achieve performance comparable to LLMs on specific tasks. While SLMs can match LLMs on domain-specific tasks, their limited reasoning abilities lead to suboptimal performance in complex CUDA generation according to our experiments. To bridge this gap, we propose ReGraphT, a training-free, retrieval-augmented generation framework that transfers LLM-level reasoning to smaller models. ReGraphT organizes CUDA optimization trajectories into a structured reasoning graph, modeling the combined CUDA optimizations as state transitions, and leverages Monte Carlo Graph Search (MCGS) for efficient exploration. We also present a CUDA-specific benchmark with difficulty tiers defined by reasoning complexity to evaluate models more comprehensively. Experiments show that ReGraphT outperforms HPC-specific fine-tuned models and other retrieval-augmented approaches, achieving an average 2.33X speedup on CUDAEval and ParEval. When paired with DeepSeek-Coder-V2-Lite-Instruct and Qwen2.5-Coder-7B-Instruct, ReGraphT enables SLMs to approach LLM-level performance without the associated privacy risks or excessive computing overhead.