mlsanityData to 14 Oct 2026

Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions

arXiv 2603.19335 ↗First posted 19 Mar 2026cs.LG cs.AIcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Post-training alignment has produced dozens of competing algorithms -- DPO, SimPO, KTO, GRPO, and others -- yet practitioners lack controlled comparisons to guide algorithm selection. We present OXRL, a unified framework implementing 51 post-training algorithms with identical infrastructure, enabling the first large-scale apples-to-apples evaluation. Our study spans 8 algorithms across 4 model scales (0.5B--7B), 3 evaluation domains, and a 20-variant DPO taxonomy (100 runs at 1.5B, 5 seeds each), totaling ~240 training runs on H100 GPUs. Three headline findings emerge. (1)~Algorithm rankings are unstable across scale: at 1.5B, online RL (SGRPO) tops all methods at 58.0%~±0.57 on GSM8K; by 7B, the worst small-scale method (SimPO) becomes the best (85.8%), a complete ranking inversion driven by model scale rather than LoRA regularization (confirmed via 2×2 factorial). (2)~Loss function modifications yield negligible gains: none of 20 DPO variants significantly outperform vanilla DPO after Bonferroni correction; the sole significant outlier, SimPO, is worse (-11.5~pp, p < 10-4). (3)~Algorithm leverage is task-specific: the 19.3~pp GSM8K spread collapses to 0.54~pp on MATH (36×) and 0.47~pp on general-domain benchmarks (41×), confirming that algorithm choice matters primarily within the training distribution. These findings yield a hierarchy of leverage for practitioners: model scale (~50~pp) gg training paradigm (~10~pp) gg online vs. offline (~9~pp) gg loss function (~1~pp). We release all code, configs, and evaluation data as a living community benchmark.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 22 of its 30 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

22 of 30 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2024Qwen2.5 Technical Reportcited by 4,942
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
10 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  3. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  5. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  6. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  9. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Taskscited by 35
  10. 2025It Takes Two: Your GRPO Is Secretly DPOcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2
  2. 2026Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training
  2. 2026What Will Post-Training Fix? Per-Problem Gains Are Shared Across Independent RL Runs, and Existing Checkpoints Predict Them Better Than A Priori Signals
  3. 2026JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
  4. 2026Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training
  5. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
  6. 2026When Offline Selectors Cannot Beat the Best Single Model: A Diagnostic Study on edX Dropout Prediction
  7. 2026WildTrace: Evaluating Training Data Detection in Realistic Large-Scale LLM Post-Training
  8. 2025Ascent Fails to Forget