mlsanityData to 14 Oct 2026

Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generation

arXiv 2512.06690 ↗First posted 7 Dec 2025cs.CLcited by 30

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Preference alignment has enabled large language models (LLMs) to better reflect human expectations, but current methods mostly optimize for population-level preferences, overlooking individual users. Personalization is essential, yet early approaches-such as prompt customization or fine-tuning-struggle to reason over implicit preferences, limiting real-world effectiveness. Recent "think-then-generate" methods address this by reasoning before response generation. However, they face challenges in long-form generation: their static one-shot reasoning must capture all relevant information for the full response generation, making learning difficult and limiting adaptability to evolving content. To address this issue, we propose FlyThinker, an efficient "think-while-generating" framework for personalized long-form generation. FlyThinker employs a separate reasoning model that generates latent token-level reasoning in parallel, which is fused into the generation model to dynamically guide response generation. This design enables reasoning and generation to run concurrently, ensuring inference efficiency. In addition, the reasoning model is designed to depend only on previous responses rather than its own prior outputs, which preserves training parallelism across different positions-allowing all reasoning tokens for training data to be produced in a single forward pass like standard LLM training, ensuring training efficiency. Extensive experiments on real-world benchmarks demonstrate that FlyThinker achieves better personalized generation while keeping training and inference efficiency. Our code is available at https://github.com/wcb0219-sketch/FlyThinker.git.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 53 references that are in the map. First posted on arXiv 7 Dec 2025.

Builds on

48 of 53 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Tagging the Thought: Unlocking Personalization Reasoning via Reinforcement Learning27 Sep
  2. 2025RPM: Reasoning-Level Personalization for Black-Box Large Language Models27 May
  3. 2025Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signals23 May
  4. 2025Reasoning Meets Personalization: Unleashing the Potential of Large Reasoning Model for Personalized Generation23 May
  5. 2025Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation7 Jan
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  7. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  8. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  9. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  10. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  11. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 99
  12. 2024LongLaMP: A Benchmark for Personalized Long-form Text Generationcited by 97
29 more
  1. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 92
  2. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 90
  3. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 74
  4. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 69
  5. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 60
  6. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 60
  7. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  8. 2025Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendationcited by 51
  9. 2025Reinforced Latent Reasoning for LLM-based Recommendationcited by 47
  10. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 45
  11. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  12. 2024Harnessing Large Language Models for Text-Rich Sequential Recommendationcited by 38
  13. 2023Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Trainingcited by 35
  14. 2025LaMP-QA: A Benchmark for Personalized Long-form Question Answeringcited by 31
  15. 2025NextQuill: Causal Preference Modeling for Enhancing LLM Personalizationcited by 30
  16. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  17. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 27
  18. 2025ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generationcited by 26
  19. 2024Large Language Models are Learnable Planners for Long-Term Recommendationcited by 24
  20. 2025PROPER: A Progressive Learning Framework for Personalized Large Language Models with Group-Level Adaptationcited by 24
  21. 2024Context Steering: Controllable Personalization at Inference Timecited by 22
  22. 2025Latent Inter-User Difference Modeling for LLM Personalizationcited by 21
  23. 2025Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Informationcited by 17
  24. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferencescited by 13
  25. 2025Aligning LLMs by Predicting Preferences from User Writing Samplescited by 11
  26. 2025DEEPER Insight into Your User: Directed Persona Refinement for Dynamic Persona Modelingcited by 10
  27. 2025SteerX: Disentangled Steering for LLM Personalizationcited by 7
  28. 2024Leveraging Memory Retrieval to Enhance LLM-based Generative Recommendationcited by 6
  29. 2025Decoding in Latent Spaces for Efficient Inference in LLM-based Recommendationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 30.

30 papers.

Sort:

By area:

  1. 2026Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoningcited by 25
  2. 2026ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selectioncited by 18
  3. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Taskscited by 9
  4. 2026Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimizationcited by 8
  5. 2026FBS: Modeling Native Parallel Reading inside a Transformercited by 7
  6. 2026Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generationcited by 5
  7. 2026PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Modelscited by 5
  8. 2026Self-Correcting RAG: Enhancing Faithfulness via MMKP Context Selection and NLI-Guided MCTScited by 5
  9. 2026Medical Reasoning with Large Language Models: A Survey and MR-Benchcited by 4
  10. 2026FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoningcited by 4
  11. 2026Learning to Control Summaries with Score Rankingcited by 4
  12. 2026Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Loopingcited by 3
18 more
  1. 2026RCBSF: A Multi-Agent Framework for Automated Contract Revision via Stackelberg Gamecited by 3
  2. 2026Know You Before You Speak: User-State Modeling for LLM Personalization in Multi-Turn Conversationcited by 3
  3. 2026Preference-Aware Rubric Learning for Personalized Evaluationcited by 2
  4. 2026AGSC: Adaptive Granularity and Semantic Clustering for Uncertainty Quantification in Long-text Generationcited by 1
  5. 2026CAP-CoT: Cycle Adversarial Prompt for Improving Chain of Thoughts in LLM Reasoningcited by 1
  6. 2026From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priorscited by 1
  7. 2026Towards Root Memories: Benchmarking and Enhancing Implicit Logical Memory Retrieval for Personalized LLMscited by 1
  8. 2026Agent-GWO: Collaborative Agents for Dynamic Prompt Optimization in Large Language Modelscited by 0
  9. 2026Small Language Model Helps Resolve Semantic Ambiguity of LLM Promptcited by 0
  10. 2026Do LLMs Capture Embodied Cognition and Cultural Variation? Cross-Linguistic Evidence from Demonstrativescited by 0
  11. 2026Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendationcited by 0
  12. 2026From Empathy to Personalized Empathy: Adapting Empathetic Strategies to Individual Userscited by 0
  13. 2026Efficient RAG with Intent-Aware Retrieval and Semantics-Preserving Chunkingcited by 0
  14. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  15. 2026Customer-Agent: Overcoming Context Limitations in Ultra-Long Shopping Trajectories via Tool-Augmented Agents and RLVRcited by 0
  16. 2026Intuition-Guided Latent Reasoning for LLM-Based Recommendationcited by 0
  17. 2026Rubric-Aware On-Policy Self-Distillation for LLM Personalizationcited by 0
  18. 2026PersonaFactor: Preference Factors with Dynamic Selection for LLM Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pathways of Thoughts: Multi-Directional Thinking for Long-form Personalized Question Answering
  2. 2026S2GR: Stepwise Semantic-Guided Reasoning in Latent Space for Generative Recommendation
  3. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
  4. 2025Aligning Deep Implicit Preferences by Learning to Reason Defensively
  5. 2024PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking
  6. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  7. 2026Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data
  8. 2026Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibration