# Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation

arXiv: https://arxiv.org/abs/2501.04167 · HTML: https://mlsanity.com/paper/2501.04167/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 46 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 16 of 16)

- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [Hello Again! LLM-powered Personalized Agent for Long-term Dialogue](https://mlsanity.com/paper/2406.05925/) (2024)
- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Self-Training: A Survey](https://mlsanity.com/paper/2202.12040/) (2022)
- [Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts](https://mlsanity.com/paper/2406.10471/) (2024)
- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [Teach LLMs to Personalize -- An Approach inspired by Writing Education](https://mlsanity.com/paper/2308.07968/) (2023)
- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models](https://mlsanity.com/paper/2409.09510/) (2024)
- [Corporate Communication Companion (CCC): An LLM-empowered Writing Assistant for Workplace Social Media](https://mlsanity.com/paper/2405.04656/) (2024)
- [A Survey on Large Language Models for Personalized and Explainable Recommendations](https://mlsanity.com/paper/2311.12338/) (2023)
- [Automated Evaluation of Personalized Text Generation using Large Language Models](https://mlsanity.com/paper/2310.11593/) (2023)

## Built on by (most cited first, 27 of 27)

- [A Survey of Personalized Large Language Models: Progress and Future Directions](https://mlsanity.com/paper/2502.11528/) (2025)
- [Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalization](https://mlsanity.com/paper/2503.02450/) (2025)
- [LaMP-QA: A Benchmark for Personalized Long-form Question Answering](https://mlsanity.com/paper/2506.00137/) (2025)
- [Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generation](https://mlsanity.com/paper/2512.06690/) (2025)
- [A Survey of Personalization: From RAG to Agent](https://mlsanity.com/paper/2504.10147/) (2025)
- [ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation](https://mlsanity.com/paper/2501.14956/) (2025)
- [Latent Inter-User Difference Modeling for LLM Personalization](https://mlsanity.com/paper/2507.20849/) (2025)
- [Instant Personalized Large Language Model Adaptation via Hypernetwork](https://mlsanity.com/paper/2510.16282/) (2025)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approach](https://mlsanity.com/paper/2505.18882/) (2025)
- [A Cognitive Writing Perspective for Constrained Long-Form Text Generation](https://mlsanity.com/paper/2502.12568/) (2025)
- [Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directions](https://mlsanity.com/paper/2602.22680/) (2026)
- [LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning](https://mlsanity.com/paper/2506.18841/) (2025)
- [Can LLMs Simulate Social Media Engagement? A Study on Action-Guided Response Generation](https://mlsanity.com/paper/2502.12073/) (2025)
- [SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models](https://mlsanity.com/paper/2506.04180/) (2025)
- [Learning from Natural Language Feedback for Personalized Question Answering](https://mlsanity.com/paper/2508.10695/) (2025)
- [Pathways of Thoughts: Multi-Directional Thinking for Long-form Personalized Question Answering](https://mlsanity.com/paper/2509.19094/) (2025)
- [Tagging the Thought: Unlocking Personalization Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2509.23140/) (2025)
- [Rethinking Personalization in Large Language Models at the Token Level](https://mlsanity.com/paper/2603.06595/) (2026)
- [Preference-Aware Rubric Learning for Personalized Evaluation](https://mlsanity.com/paper/2605.31545/) (2026)
- [Improving User Privacy in Personalized Generation: Client-Side Retrieval-Augmented Modification of Server-Side Generated Speculations](https://mlsanity.com/paper/2601.17569/) (2026)
- [Reasoning-Based Personalized Generation for Users with Sparse Data](https://mlsanity.com/paper/2602.21219/) (2026)
- [Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.07794/) (2025)
- [Sparse Personalized Text Generation with Multi-Trajectory Reasoning](https://mlsanity.com/paper/2604.24996/) (2026)
- [Psi-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues](https://mlsanity.com/paper/2606.02754/) (2026)
- [Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs](https://mlsanity.com/paper/2606.09038/) (2026)
- [LogicTree-RAG: Logic Tree-guided Retrieval-Augmented Generation for Long-form Patent Drafting](https://mlsanity.com/paper/2609.30943/) (2026)

## Abstract (arXiv)

Personalized text generation requires a unique ability of large language models (LLMs) to learn from context that they often do not encounter during their standard training. One way to encourage LLMs to better use personalized context for generating outputs that better align with the user's expectations is to instruct them to reason over the user's past preferences, background knowledge, or writing style. To achieve this, we propose Reasoning-Enhanced Self-Training for Personalized Text Generation (REST-PG), a framework that trains LLMs to reason over personal data during response generation. REST-PG first generates reasoning paths to train the LLM's reasoning abilities and then employs Expectation-Maximization Reinforced Self-Training to iteratively train the LLM based on its own high-reward outputs. We evaluate REST-PG on the LongLaMP benchmark, consisting of four diverse personalized long-form text generation tasks. Our experiments demonstrate that REST-PG achieves significant improvements over state-of-the-art baselines, with an average relative performance gain of 14.5% on the benchmark.