mlsanityData to 13 Oct 2026

Risk-Averse Finetuning of Large Language Models

arXiv 2501.06911 ↗First posted 12 Jan 2025cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

We consider the challenge of mitigating the generation of negative or toxic content by the Large Language Models (LLMs) in response to certain prompts. We propose integrating risk-averse principles into LLM fine-tuning to minimize the occurrence of harmful outputs, particularly rare but significant events. By optimizing the risk measure of Conditional Value at Risk (CVaR), our methodology trains LLMs to exhibit superior performance in avoiding toxic outputs while maintaining effectiveness in generative tasks. Empirical evaluations on sentiment modification and toxicity mitigation tasks demonstrate the efficacy of risk-averse reinforcement learning with human feedback (RLHF) in promoting a safer and more constructive online discourse environment.

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 12 Jan 2025.

Builds on

43 of 55 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2022Training language models to follow instructions with human feedbackcited by 15,856
  5. 2017Proximal Policy Optimization Algorithmscited by 14,592
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  11. 2017Deep reinforcement learning from human preferencescited by 3,871
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
31 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,586
  3. 2020Learning to summarize from human feedbackcited by 2,575
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,139
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,736
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,249
  7. 2021Ethical and social risks of harm from Language Modelscited by 964
  8. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 723
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 718
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 648
  11. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  12. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 437
  13. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  14. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 363
  15. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  16. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  17. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  18. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  19. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  20. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  21. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  22. 2015Policy Gradient for Coherent Risk Measurescited by 75
  23. 2019Worst Cases Policy Gradientscited by 52
  24. 2022Efficient Risk-Averse Reinforcement Learningcited by 41
  25. 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
  26. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 28
  27. 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 19
  28. 2019Learning Robust Options by Conditional Value at Risk Optimizationcited by 16
  29. 2023Systematic Rectification of Language Models via Dead-end Analysiscited by 16
  30. 2018A Block Coordinate Ascent Algorithm for Mean-Variance Optimizationcited by 11
  31. 2021Policy Gradient Methods for Distortion Risk Measurescited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2026Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuningcited by 6
  2. 2025Knowledge Graph-Infused Fine-Tuning for Structured Reasoning in Large Language Modelscited by 5
  3. 2026AI Alignment Breaks at the Edgecited by 3
  4. 2025Risk-aware Direct Preference Optimization under Nested Risk Measurecited by 2
  5. 2025ESLM: Risk-Averse Selective Language Modeling for Efficient Pretrainingcited by 1
  6. 2025Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inferencecited by 0
  7. 2026Fine-Tuning Generative Models for Extreme Events via CVaR-Penalized Wasserstein Gradient Flowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Risk-Conditioned Fine-Tuning of Large Language Models
  2. 2026Characterising Toxicity in Generative Large Language Models
  3. 2026Measuring and Mitigating Toxicity in Large Language Models: A Comprehensive Replication Study
  4. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
  5. 2024Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF
  6. 2022Reward Modeling for Mitigating Toxicity in Transformer-based Language Models
  7. 2024Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation
  8. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs