mlsanityData to 14 Oct 2026

SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models

arXiv 2509.24781 ↗First posted 29 Sep 2025cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Existing alignment methods for preference optimization of large language models (LLMs) aim to enhance model performance by utilizing pairs of positive and negative samples. However, due to the limited capacity of models in scoring or generating responses, the quality of positive and negative samples may become similar during training, which complicates optimization for preference learning. To address this issue, we introduce SeaPO, a Strategic Error Amplification method that leverages three error types commonly occurring in LLMs to introduce specific error patterns into the model Preference Optimization. This strategy ensures that negative samples are more erroneous than positive samples and preference-based training is employed to mitigate the occurrence of these errors, thereby enhancing model performance. Evaluations across five capability dimensions and different model scales (1.5B to 14B) demonstrate that the generated data significantly improved overall model performance, particularly in terms of truthfulness, with improvements of 5-10 percentage points observed. Further analysis reveals that task performance varies depending on the error types introduced. Injecting the most common error types improves performance in related tasks, while a mix of error types leads to a broader performance enhancement: most tasks show stable improvements, while a few tasks exhibit significant gains.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 55 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

35 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Self-Boosting Large Language Models with Synthetic Preference Data9 Oct
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  4. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  5. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning7 Feb

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
18 more
  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  3. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  5. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  6. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  7. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  8. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  9. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  10. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  11. 2024FunnelRAG: A Coarse-to-Fine Progressive Retrieval Paradigm for RAGcited by 18
  12. 2024CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitionscited by 14
  13. 2025APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Trainingcited by 12
  14. 2025Doubling Your Data in Minutes: Ultra-fast Tabular Data Generation via LLM-Induced Dependency Graphscited by 9
  15. 2024NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updatescited by 8
  16. 2025Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Modelscited by 8
  17. 2024Length-Aware Multi-Kernel Transformer for Long Document Classificationcited by 5
  18. 2024Medico: Towards Hallucination Detection and Correction with Multi-source Evidence Fusioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ROPO: Robust Preference Optimization for Large Language Models
  2. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  3. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  4. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
  5. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
  6. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
  7. 2025ComPO: Preference Alignment via Comparison Oracles
  8. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge