# Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuning

arXiv: https://arxiv.org/abs/2503.22456 · HTML: https://mlsanity.com/paper/2503.22456/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning](https://mlsanity.com/paper/2405.00451/) (2024)
- [Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models](https://mlsanity.com/paper/2412.15287/) (2024)
- [Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search](https://mlsanity.com/paper/2405.15383/) (2024)
- [Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning](https://mlsanity.com/paper/2502.06533/) (2025)
- [PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replay](https://mlsanity.com/paper/2112.03798/) (2021)
- [Average-Reward Soft Actor-Critic](https://mlsanity.com/paper/2501.09080/) (2025)
- [Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning](https://mlsanity.com/paper/2306.15503/) (2023)

## Built on by (most cited first, 13 of 13)

- [Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents](https://mlsanity.com/paper/2509.09265/) (2025)
- [Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning](https://mlsanity.com/paper/2508.02260/) (2025)
- [EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control](https://mlsanity.com/paper/2511.15248/) (2025)
- [Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought](https://mlsanity.com/paper/2603.22847/) (2026)
- [TableMoE: Neuro-Symbolic Routing for Structured Expert Reasoning in Multimodal Table Understanding](https://mlsanity.com/paper/2506.21393/) (2025)
- [Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models](https://mlsanity.com/paper/2510.09259/) (2025)
- [Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning](https://mlsanity.com/paper/2512.04359/) (2025)
- [When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy](https://mlsanity.com/paper/2605.12112/) (2026)
- [The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Sampling](https://mlsanity.com/paper/2510.15502/) (2025)
- [Distribution-Centric Policy Optimization Dominates Exploration-Exploitation Trade-off](https://mlsanity.com/paper/2601.12730/) (2026)
- [InstructDiff: Domain-Adaptive Data Selection via Differential Entropy for Efficient LLM Fine-Tuning](https://mlsanity.com/paper/2601.23006/) (2026)
- [Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization](https://mlsanity.com/paper/2605.08978/) (2026)
- [GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMs](https://mlsanity.com/paper/2511.13007/) (2025)

## Abstract (arXiv)

We introduce Entropy-Guided Sequence Weighting (EGSW), a novel approach that enhances the exploration-exploitation tradeoff by dynamically assigning weights to generated outputs based on their advantage and entropy for Reinforcement Learning-based Large Language Model fine-tuning. EGSW integrates entropy regularization with advantage-based weighting to balance policy updates, enabling efficient exploration in high-dimensional state spaces. By employing temperature-scaled softmax weighting over sequences, EGSW prioritizing high-reward, high-uncertainty steps while maintaining training stability. Although originally developed to improve Group Relative Policy Optimization (GRPO) during large language model (LLM) fine-tuning, EGSW is generalizable to other reinforcement learning (RL) algorithms and can be implemented in both step-wise and trajectory-wise settings. Empirical evaluations demonstrate that EGSW enhances GRPO reasoning ability, yielding improvements in sample efficiency. Future work will explore the application of EGSW to advanced RL methodologies.