mlsanityData to 7 Oct 2026

Risk-Conditioned Fine-Tuning of Large Language Models

arXiv 2609.08064 ↗First posted 8 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed in settings where rare but severe harmful generations can have significant consequences. Existing Risk-Averse RLHF addresses this issue by optimizing Conditional Value-at-Risk (CVaR), but it trains policies for fixed risk levels and therefore cannot adjust the desired degree of risk aversion at inference time. In this paper, we propose risk-conditioned RLHF, a framework that trains a single policy that provides a continuous risk-control interface, enabling users to select different degrees of risk aversion without retraining or deploying multiple risk-specific models. Experiments across multiple benchmarks demonstrate that a single risk-conditioned policy can adapt to different risk levels at inference time, enabling more flexible and risk-aware LLM deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 80 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

42 of 80 references.

References · 42

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
30 more
  1. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  2. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  3. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  4. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  5. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  6. 2018Stochastic model-based minimization of weakly convex functionscited by 157
  7. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  8. 2013Risk-sensitive Reinforcement Learningcited by 129
  9. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  10. 2014Optimizing the CVaR via Samplingcited by 98
  11. 2014Algorithms for CVaR Optimization in MDPscited by 98
  12. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  13. 2015Policy Gradient for Coherent Risk Measurescited by 74
  14. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  15. 2016Efficiency of minimizing compositions of convex functions and smooth mapscited by 59
  16. 2023The Next Chapter: A Study of Large Language Models in Storytellingcited by 58
  17. 2024Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Modelscited by 51
  18. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  19. 2023DisasterResponseGPT: Large Language Models for Accelerated Plan of Action Development in Disaster Response Scenarioscited by 33
  20. 2018Stochastic subgradient method converges at the rate O(k-1/4) on weakly convex functionscited by 31
  21. 2021Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigationcited by 13
  22. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  23. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniquescited by 9
  24. 2021Two steps to risk sensitivitycited by 8
  25. 2025Alignment of large language models with constrained learningcited by 8
  26. 2025From Prompts to Protection: Large Language Model-Enabled In-Context Learning for Smart Public Safety UAVcited by 8
  27. 2024Integration of Large Vision Language Models for Efficient Post-disaster Damage Assessment and Reportingcited by 6
  28. 2024Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processescited by 4
  29. 2024Continuous Language Model Interpolation for Dynamic and Controllable Text Generationcited by 2
  30. 2026Robust General Utility for Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Risk-Averse Finetuning of Large Language Models
  2. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models
  3. 2024Evaluating Defences against Unsafe Feedback in RLHF
  4. 2026From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuning
  5. 2025Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignment
  6. 2026Robust Policy Optimization to Prevent Catastrophic Forgetting
  7. 2025Risk-aware Direct Preference Optimization under Nested Risk Measure
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback