mlsanityData to 14 Oct 2026

ProSocialAlign: Preference Conditioned Test Time Alignment in Language Models

arXiv 2512.06515 ↗First posted 6 Dec 2025cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Current language model safety paradigms often fall short in emotionally charged or high-stakes settings, where refusal-only approaches may alienate users and naive compliance can amplify risk. We propose ProSocialAlign, a test-time, parameter-efficient framework that steers generation toward safe, empathetic, and value-aligned responses without retraining the base model. We formalize five human-centered objectives and cast safety as lexicographic constrained generation: first, applying hard constraints to eliminate harmful continuations; then optimizing for prosocial quality within the safe set. Our method combines (i) directional regulation, a harm-mitigation mechanism that subtracts a learned "harm vector" in parameter space, and (ii) preference-aware autoregressive reward modeling trained jointly across attributes with gradient conflict resolution, enabling fine-grained, user-controllable decoding. Empirical evaluations across five safety benchmarks demonstrate state-of-the-art performance, reducing unsafe leakage and boosting alignment to human values, with strong gains across multiple evaluation metrics. ProSocialAlign offers a robust and modular foundation for generating context-sensitive, safe, and human-aligned responses at inference time.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 85 references that are in the map. First posted on arXiv 6 Dec 2025.

Builds on

43 of 85 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model6 May
  2. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment10 Oct
  3. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  4. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  5. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  6. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  7. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 35

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  8. 2022Editing Models with Task Arithmeticcited by 1,120
  9. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  10. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
23 more
  1. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  3. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  4. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  5. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  6. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  7. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  8. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  9. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  10. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  11. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  12. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  13. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  14. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  15. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  16. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activationscited by 32
  17. 2024SafeInfer: Context Adaptive Decoding Time Safety Alignment for Large Language Modelscited by 23
  18. 2024RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Evictioncited by 21
  19. 2024Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Modelscited by 20
  20. 2024Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Mergingcited by 14
  21. 2025Learning Safety Constraints for Large Language Modelscited by 13
  22. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
  23. 2024Thresholded Lexicographic Ordered Multiobjective Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Structurally-bounded Agentic Graph Exploration for Evidence-Grounded Scholarly DeepSearchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  2. 2026Cat-DPO: Category-Adaptive Safety Alignment
  3. 2024Negative-Prompt-driven Alignment for Generative Language Model
  4. 2026Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retraining
  5. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
  6. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  7. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  8. 2026Pressure Reveals Character: Behavioural Alignment Evaluation at Depth