# Improving LLM Safety and Helpfulness using SFT and DPO: A Study on OPT-350M

arXiv: https://arxiv.org/abs/2509.09055 · HTML: https://mlsanity.com/paper/2509.09055/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 1 of its 2 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 1 of 1)

- [Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks](https://mlsanity.com/paper/2404.14723/) (2024)

## Built on by (most cited first, 3 of 3)

- [Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models](https://mlsanity.com/paper/2510.12044/) (2025)
- [AFRILANGTUTOR: Advancing Language Tutoring and Culture Education in Low-Resource Languages with Large Language Models](https://mlsanity.com/paper/2604.20996/) (2026)
- [Equity-Aware Geospatial AI for Forecasting Demand-Driven Hospital Locations in Germany](https://mlsanity.com/paper/2510.10640/) (2025)

## Abstract (arXiv)

This research investigates the effectiveness of alignment techniques, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and a combined SFT+DPO approach on improving the safety and helpfulness of the OPT-350M language model. Utilizing the Anthropic Helpful-Harmless RLHF dataset, we train and evaluate four models: the base OPT350M, an SFT model, a DPO model, and a model trained with both SFT and DPO. We introduce three key evaluation metrics: Harmlessness Rate (HmR), Helpfulness Rate (HpR), and a Combined Alignment Score (CAS), all derived from reward model outputs. The results show that while SFT outperforms DPO, The combined SFT+DPO model outperforms all others across all metrics, demonstrating the complementary nature of these techniques. Our findings also highlight challenges posed by noisy data, limited GPU resources, and training constraints. This study offers a comprehensive view of how fine-tuning strategies affect model alignment and provides a foundation for more robust alignment pipelines in future work.