mlsanityData to 14 Oct 2026

Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints

arXiv 2508.08466 ↗First posted 11 Aug 2025cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Small large language models (LLMs) often face difficulties in aligning output to human preferences, particularly when operating under severe performance gaps. In this work, we propose two lightweight DPO-based variants -- Adaptive Margin-Sigmoid Loss and APO-hinge-zero -- to better address underperformance scenarios by introducing margin-based objectives and selective update mechanisms. Our APO-hinge-zero method, which combines hinge-induced hard-example mining with the chosen-focused optimization of APO-zero, achieves strong results. In AlpacaEval, APO-hinge-zero improves the win rate by +2.0 points and the length-controlled win rate by +1.4 points compared to the APO-zero baseline. In MT-Bench, our methods maintain competitive performance in diverse categories, particularly excelling in STEM and Humanities tasks. These results demonstrate that simple modifications to preference-based objectives can significantly enhance small LLM alignment under resource constraints, offering a practical path toward more efficient deployment.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 16 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

13 of 16 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  2. 2024Distributional Preference Alignment of LLMs via Optimal Transport9 Jun
  3. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  4. 2024Towards Efficient Exact Optimization of Language Model Alignment1 Feb
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2023LIMA: Less Is More for Alignment18 May
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 6

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Provably Convergent Primal-Dual DPO for Constrained LLM Alignment
  2. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  3. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  4. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
  5. 2026MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment
  6. 2026EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent
  7. 2024Sample-Efficient Alignment for LLMs
  8. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples