mlsanityData to 7 Oct 2026

Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference

arXiv 2510.21184 ↗First posted 24 Oct 2025cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) has become a predominant technique to align language models (LMs) with human preferences or promote outputs which are deemed to be desirable by a given reward function. Standard RL approaches optimize average reward, while methods explicitly focused on reducing the probability of undesired outputs typically come at a cost to average-case performance. To improve this tradeoff, we introduce RePULSe, a new training method that augments the standard RL loss with an additional loss that uses learned proposals to guide sampling low-reward outputs, and then reduces those outputs' probability. We run experiments demonstrating that RePULSe produces a better tradeoff of expected reward versus the probability of undesired outputs and is more adversarially robust, compared to standard RL alignment approaches and alternatives.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 60 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

48 of 60 references.

References · 48

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2015Prioritized Experience Replaycited by 1,070
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
36 more
  1. 2022Red Teaming Language Models with Language Modelscited by 983
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  5. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  7. 2023Towards Unbounded Machine Unlearningcited by 261
  8. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  9. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  10. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  11. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  12. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  13. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  14. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  15. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  16. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 79
  17. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 57
  18. 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 42
  19. 2022Efficient Risk-Averse Reinforcement Learningcited by 41
  20. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  21. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  22. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  23. 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
  24. 2019Negative Training for Neural Dialogue Response Generationcited by 24
  25. 2024Detecting Mode Collapse in Language Models via Narrationcited by 24
  26. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
  27. 2019Distributional Reinforcement Learning for Energy-Based Sequential Modelscited by 17
  28. 2022A2C is a special case of PPOcited by 16
  29. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  30. 2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedbackcited by 13
  31. 2022BR-SNIS: Bias Reduced Self-Normalized Importance Samplingcited by 12
  32. 2024Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policiescited by 8
  33. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimizationcited by 7
  34. 2025Risk-Averse Finetuning of Large Language Modelscited by 7
  35. 2018Detecting egregious responses in neural sequence-to-sequence modelscited by 6
  36. 2023Value-aware Importance Weighting for Off-policy Reinforcement Learningcited by 4

Built on by

Not cited yet.