# From Reasoning to Code: GRPO Optimization for Underrepresented Languages

arXiv: https://arxiv.org/abs/2506.11027 · HTML: https://mlsanity.com/paper/2506.11027/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://mlsanity.com/paper/2501.04519/) (2025)
- [RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning](https://mlsanity.com/paper/2410.02089/) (2024)
- [Arithmetic Reasoning with LLM: Prolog Generation & Permutation](https://mlsanity.com/paper/2405.17893/) (2024)
- [Program Synthesis Through Reinforcement Learning Guided Tree Search](https://mlsanity.com/paper/1806.02932/) (2018)

## Built on by (most cited first, 11 of 11)

- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions](https://mlsanity.com/paper/2602.20213/) (2026)
- [Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs](https://mlsanity.com/paper/2510.04140/) (2025)
- [StaRPO: Stability-Augmented Reinforcement Policy Optimization](https://mlsanity.com/paper/2604.08905/) (2026)
- [Find, Fix, Reason: Context Repair for Video Reasoning](https://mlsanity.com/paper/2604.16243/) (2026)
- [Structured Reasoning for Large Language Models](https://mlsanity.com/paper/2601.07180/) (2026)
- [Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents](https://mlsanity.com/paper/2606.05296/) (2026)
- [From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training](https://mlsanity.com/paper/2607.16257/) (2026)
- [MoL-RL: Distilling Multi-Step Environmental Feedback into LLMs for Feedback-Independent Reasoning](https://mlsanity.com/paper/2507.20278/) (2025)
- [GAPO: Robust Advantage Estimation for Real-World Code LLMs](https://mlsanity.com/paper/2510.21830/) (2025)
- [Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO](https://mlsanity.com/paper/2606.05174/) (2026)

## Abstract (arXiv)

Generating accurate and executable code using Large Language Models (LLMs) remains a significant challenge for underrepresented programming languages, such as Prolog and Lisp, due to the scarcity of public training data compared to high-resource languages like Python. This paper introduces a generalizable Reinforcement Learning (RL) approach that combines small-scale versions of the Qwen2.5-Coder model with Group Relative Policy Optimization (GRPO) to enable effective code generation through reasoning. To address the limitations of sparse datasets, we integrate execution-driven feedback directly into the RL loop, utilizing a reward system that exploits both logical correctness and structural formatting. Experimental results on GSM8K dataset demonstrate significant improvements in reasoning quality and code accuracy across underrepresented languages. These findings underscore the potential of our approach to benefit a wide range of programming languages lacking extensive training resources by leveraging symbolic reasoning and interpreter-based feedback.