mlsanityData to 14 Oct 2026

ISO: An RLVR-Native Optimization Stack

arXiv 2607.19331 ↗First posted 21 Jul 2026cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) is rapidly advancing the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood. Building on our prior analysis (Zhu et al., 2025), we study this missing layer through the singular structure of model weights and identify spectral inheritance: RLVR can reuse the base model's weight spectra while acquiring new behavior through changes in the associated input and output singular frames. We operationalize spectral inheritance as Isospectral Optimization (ISO), an RLVR-native, fixed-spectrum optimization framework with complementary offline and online instantiations. Offline, ISO-Merger combines the frame changes of shared-base specialists into a single fixed-spectrum model, requiring no post-merge data, rollouts, gradient updates, or on-policy distillation (OPD). It recovers complementary specialist capabilities and achieves the strongest aggregate performance among the compared data-free merging methods. Online, ISO-Optimizer applies a chosen base optimizer, including AdamW and Muon, to the frame variables while keeping the base spectra fixed. Across reasoning and coding tasks ranging from 1.5B to 8B parameters, ISO-Optimizer improves accuracy in the reported runs and reaches matched scores with substantially fewer training steps. On Qwen3-8B-Base, AdamW reaches an aggregate accuracy of 0.495 after 270 training steps. ISO-AdamW reaches the same accuracy after only 100 training steps and improves further to 0.509 after 210 training steps. Together, ISO offers a concrete answer to RLVR's missing optimization layer: rather than inheriting pre-training optimization wholesale, design post-training around the structure of reward-driven adaptation: inherit the spectrum, optimize the frames.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 41 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

31 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025The Path Not Taken: RLVR Provably Learns Off the Principals11 Nov
  2. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration1 Oct
  3. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR21 Jul
  4. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan

Other references · 25

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  3. 2022Editing Models with Task Arithmeticcited by 1,120
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  5. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  6. 2025Muon is Scalable for LLM Trainingcited by 428
  7. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  8. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  9. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  10. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  11. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
  12. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
13 more
  1. 2025Dion: Distributed Orthonormalized Updatescited by 68
  2. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 64
  3. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 56
  4. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
  5. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
  6. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  7. 2026Maximum Likelihood Reinforcement Learningcited by 36
  8. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  9. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 14
  10. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 9
  11. 2026Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformationcited by 6
  12. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 5
  13. 2026Orthogonal Model Mergingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Trainingcited by 0
  2. 2026When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards
  2. 2026Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
  3. 2026Linear Dynamics in the RLVR Training of Large Language Models
  4. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholds
  5. 2025Evaluating Parameter Efficient Methods for RLVR
  6. 2026The Unlearnability Phenomenon in RLVR for Language Models
  7. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
  8. 2025Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards