# CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment

arXiv: https://arxiv.org/abs/2403.16649 · HTML: https://mlsanity.com/paper/2403.16649/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [Preference Ranking Optimization for Human Alignment](https://mlsanity.com/paper/2306.17492/) (2023)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Secrets of RLHF in Large Language Models Part I: PPO](https://mlsanity.com/paper/2307.04964/) (2023)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [Chain of Hindsight Aligns Language Models with Feedback](https://mlsanity.com/paper/2302.02676/) (2023)
- [Calibrating Sequence likelihood Improves Conditional Language Generation](https://mlsanity.com/paper/2210.00045/) (2022)
- [Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning](https://mlsanity.com/paper/2306.03350/) (2023)

## Built on by (most cited first, 3 of 3)

- [Safety Must Precede the Deployment of Open-Ended AI](https://mlsanity.com/paper/2502.04512/) (2025)
- [Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity](https://mlsanity.com/paper/2505.23923/) (2025)
- [DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment](https://mlsanity.com/paper/2604.01787/) (2026)

## Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a crucial technique in aligning large language models (LLMs) with human preferences, ensuring these LLMs behave in beneficial and comprehensible ways to users. However, a longstanding challenge in human alignment techniques based on reinforcement learning lies in their inherent complexity and difficulty in training. To address this challenge, we present a simple yet effective Contrastive Learning Framework for Human Alignment (CLHA) to align LLMs with human preferences directly. CLHA employs a novel rescoring strategy to evaluate the noise within the data by considering its inherent quality and dynamically adjusting the training process. Simultaneously, CLHA utilizes pairwise contrastive loss and adaptive supervised fine-tuning loss to adaptively modify the likelihood of generating responses, ensuring enhanced alignment with human preferences. Using advanced methods, CLHA surpasses other algorithms, showcasing superior performance in terms of reward model scores, automatic evaluations, and human assessments on the widely used ``Helpful and Harmless'' dataset.