mlsanityData to 7 Oct 2026

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

arXiv 2607.26515 ↗First posted 29 Jul 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout-training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B and Qwen2.5-Math-7B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 4.9% to 1.1%, bringing fully quantized FP4 RL within striking distance of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 57 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

47 of 57 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
35 more
  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  2. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  3. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  5. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  7. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  8. 2021Accelerating Sparse Deep Neural Networkscited by 213
  9. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  10. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  11. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  12. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  13. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
  14. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  15. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  16. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Modelscited by 81
  17. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  18. 2022An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformerscited by 28
  19. 2021Pareto-Optimal Quantized ResNet Is Mostly 4-bitcited by 26
  20. 2025FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Modelscited by 26
  21. 2025REARANK: Reasoning Re-ranking Agent via Reinforcement Learningcited by 24
  22. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMscited by 23
  23. 2021S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Accelerationcited by 19
  24. 2024Accelerating Transformer Pre-training with 2:4 Sparsitycited by 18
  25. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  26. 2025From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learningcited by 15
  27. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  28. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMscited by 14
  29. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 13
  30. 2023PDP: Parameter-free Differentiable Pruning is All You Needcited by 12
  31. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  32. 2025Accelerating Transformer Inference and Training with 2:4 Activation Sparsitycited by 9
  33. 2026HiFloat4 Format for Language Model Inferencecited by 9
  34. 2024Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Headscited by 5
  35. 2025DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Schedulingcited by 1

Built on by

Not cited yet.