# Towards Safety and Helpfulness Balanced Responses via Controllable Large Language Models

arXiv: https://arxiv.org/abs/2404.01295 · HTML: https://mlsanity.com/paper/2404.01295/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Fine-tuning Language Models for Factuality](https://mlsanity.com/paper/2311.08401/) (2023)
- [RankGen: Improving Text Generation with Large Ranking Models](https://mlsanity.com/paper/2205.09726/) (2022)
- [A causal framework for explaining the predictions of black-box sequence-to-sequence models](https://mlsanity.com/paper/1707.01943/) (2017)
- [Controlling Style in Generated Dialogue](https://mlsanity.com/paper/2009.10855/) (2020)
- [Proximal Policy Optimization and its Dynamic Version for Sequence Generation](https://mlsanity.com/paper/1808.07982/) (2018)
- [Local Explanation of Dialogue Response Generation](https://mlsanity.com/paper/2106.06528/) (2021)
- [Improving Conditional Sequence Generative Adversarial Networks by Stepwise Evaluation](https://mlsanity.com/paper/1808.05599/) (2018)

## Built on by (most cited first, 14 of 14)

- [OR-Bench: An Over-Refusal Benchmark for Large Language Models](https://mlsanity.com/paper/2405.20947/) (2024)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models](https://mlsanity.com/paper/2410.02298/) (2024)
- [Reasoning as an Adaptive Defense for Safety](https://mlsanity.com/paper/2507.00971/) (2025)
- [You Know What I'm Saying: Jailbreak Attack via Implicit Reference](https://mlsanity.com/paper/2410.03857/) (2024)
- [EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions](https://mlsanity.com/paper/2505.23473/) (2025)
- [Position: Beyond Assistance -- Reimagining LLMs as Ethical and Adaptive Co-Creators in Mental Health Care](https://mlsanity.com/paper/2503.16456/) (2025)
- [ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AI](https://mlsanity.com/paper/2602.14135/) (2026)
- [Accelerating RLHF Training with Reward Variance Increase](https://mlsanity.com/paper/2505.23247/) (2025)
- [High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuning](https://mlsanity.com/paper/2506.04051/) (2025)
- [How Semantically Stable Are LLM Refusals? Measuring Confusion in Local Safety Boundaries](https://mlsanity.com/paper/2512.01037/) (2025)
- [A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models](https://mlsanity.com/paper/2408.16751/) (2024)
- [Hidden in the Request: Explaining Unethical LLM Compliance through Token Relevance](https://mlsanity.com/paper/2608.23264/) (2026)
- [Evolving and Detecting Multi-Turn Deception using Geometric Signatures](https://mlsanity.com/paper/2605.27671/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become easily accessible nowadays, the trade-off between safety and helpfulness can significantly impact user experience. A model that prioritizes safety will cause users to feel less engaged and assisted while prioritizing helpfulness will potentially cause harm. Possible harms include teaching people how to build a bomb, exposing youth to inappropriate content, and hurting users' mental health. In this work, we propose to balance safety and helpfulness in diverse use cases by controlling both attributes in LLM. We explore training-free and fine-tuning methods that do not require extra human annotations and analyze the challenges of controlling safety and helpfulness in LLMs. Our experiments demonstrate that our method can rewind a learned model and unlock its controllability.