# Evaluating Parameter Efficient Methods for RLVR

arXiv: https://arxiv.org/abs/2512.23165 · HTML: https://mlsanity.com/paper/2512.23165/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 29 Dec 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning](https://mlsanity.com/paper/2308.03303/) (2023)
- [MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning](https://mlsanity.com/paper/2406.09044/) (2024)
- [JustRL: Scaling a 1.5B LLM with a Simple RL Recipe](https://mlsanity.com/paper/2512.16649/) (2025)
- [The Path Not Taken: RLVR Provably Learns Off the Principals](https://mlsanity.com/paper/2511.08567/) (2025)
- [Reinforcement Learning Finetunes Small Subnetworks in Large Language Models](https://mlsanity.com/paper/2505.11711/) (2025)
- [Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Models](https://mlsanity.com/paper/2211.08682/) (2022)

## Built on by (most cited first, 12 of 12)

- [Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL](https://mlsanity.com/paper/2602.13035/) (2026)
- [Learning to Reason in 13 Parameters](https://mlsanity.com/paper/2602.04118/) (2026)
- [GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR](https://mlsanity.com/paper/2601.09361/) (2026)
- [On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters](https://mlsanity.com/paper/2606.02437/) (2026)
- [Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging](https://mlsanity.com/paper/2608.05541/) (2026)
- [BoostLoRA: Growing Effective Rank by Boosting Adapters](https://mlsanity.com/paper/2604.27308/) (2026)
- [Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines](https://mlsanity.com/paper/2605.01077/) (2026)
- [FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning](https://mlsanity.com/paper/2605.22869/) (2026)
- [The Hidden Power of Scaling Factor in LoRA Optimization](https://mlsanity.com/paper/2606.12883/) (2026)
- [Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR](https://mlsanity.com/paper/2606.31813/) (2026)
- [Normalized Low-Rank Adaptation](https://mlsanity.com/paper/2608.31036/) (2026)
- [The Low-Rank Structure of VLA Reinforcement Learning](https://mlsanity.com/paper/2609.34599/) (2026)

## Abstract (arXiv)

We systematically evaluate Parameter-Efficient Fine-Tuning (PEFT) methods under the paradigm of Reinforcement Learning with Verifiable Rewards (RLVR). RLVR incentivizes language models to enhance their reasoning capabilities through verifiable feedback; however, while methods like LoRA are commonly used, the optimal PEFT architecture for RLVR remains unidentified. In this work, we conduct the first comprehensive evaluation of over 12 PEFT methodologies across the DeepSeek-R1-Distill families on mathematical reasoning benchmarks. Our empirical results challenge the default adoption of standard LoRA with three main findings. First, we demonstrate that structural variants, such as DoRA, AdaLoRA, and MiSS, consistently outperform LoRA. Second, we uncover a spectral collapse phenomenon in SVD-informed initialization strategies (textite.g., PiSSA, MiLoRA), attributing their failure to a fundamental misalignment between principal-component updates and RL optimization. Furthermore, our ablations reveal that extreme parameter reduction (textite.g., VeRA, Rank-1) severely bottlenecks reasoning capacity. We further conduct ablation studies and scaling experiments to validate our findings. This work provides a definitive guide for advocating for more exploration for parameter-efficient RL methods.