# Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization

arXiv: https://arxiv.org/abs/2510.19325 · HTML: https://mlsanity.com/paper/2510.19325/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 15 of 15)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [BillSum: A Corpus for Automatic Summarization of US Legislation](https://mlsanity.com/paper/1910.00523/) (2019)
- [Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback](https://mlsanity.com/paper/2306.00186/) (2023)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [Summarization is (Almost) Dead](https://mlsanity.com/paper/2309.09558/) (2023)
- [Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewards](https://mlsanity.com/paper/2107.00176/) (2021)
- [The Hypervolume Indicator: Problems and Algorithms](https://mlsanity.com/paper/2005.00515/) (2020)
- [Verco: Learning Coordinated Verbal Communication for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2404.17780/) (2024)
- [Multi-Dimensional Optimization for Text Summarization via Reinforcement Learning](https://mlsanity.com/paper/2406.00303/) (2024)
- [AugSumm: towards generalizable speech summarization using synthetic labels from large language model](https://mlsanity.com/paper/2401.06806/) (2024)

## Built on by (most cited first, 3 of 3)

- [IB-GRPO: Aligning LLM-based Learning Path Recommendation with Educational Objectives via Indicator-Based Group Relative Policy Optimization](https://mlsanity.com/paper/2601.14686/) (2026)
- [Top-k Pareto Bandits: Hypervolume Regret for Multi-Objective Slate Selection](https://mlsanity.com/paper/2607.26273/) (2026)
- [TIAO: Token Importance-Aware Policy Optimization for Text Summarization](https://mlsanity.com/paper/2609.16748/) (2026)

## Abstract (arXiv)

Text summarization is a crucial task that requires the simultaneous optimization of multiple objectives, including consistency, coherence, relevance, and fluency, which presents considerable challenges. Although large language models (LLMs) have demonstrated remarkable performance, enhanced by reinforcement learning (RL), few studies have focused on optimizing the multi-objective problem of summarization through RL based on LLMs. In this paper, we introduce hypervolume optimization (HVO), a novel optimization strategy that dynamically adjusts the scores between groups during the reward process in RL by using the hypervolume method. This method guides the model's optimization to progressively approximate the pareto front, thereby generating balanced summaries across multiple objectives. Experimental results on several representative summarization datasets demonstrate that our method outperforms group relative policy optimization (GRPO) in overall scores and shows more balanced performance across different dimensions. Moreover, a 7B foundation model enhanced by HVO performs comparably to GPT-4 in the summarization task, while maintaining a shorter generation length. Our code is publicly available at https://github.com/ai4business-LiAuto/HVO.git