mlsanityData to 14 Oct 2026

SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models

arXiv 2608.06179 ↗First posted 6 Aug 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Language models for low-resource languages still make frequent grammatical errors. Alignment methods such as RLHF and DPO can correct them. However, they require human judgements, which are scarce for languages such as Icelandic. Automatic rewards offer an alternative, yet models can learn to exploit them without improving (reward hacking). We present SAGA (Score-weighted Adaptive Generation Alignment), which builds preference pairs using dependency parsers. Such parsers already exist for many low-resource languages. SAGA's reward combines parse quality with lexical diversity. It keeps only pairs whose reward gap reaches a threshold and trains on them with DPO. On Danish, Icelandic and Norwegian Bokmaal, SAGA raises reward-parser success by 18 to 28 points. It improves all four model families we test, although one Icelandic run is flagged for reward hacking. An independent parser confirms the gains, and native speakers prefer SAGA outputs in Danish (69%) and Icelandic (80%). Online RL methods exploit the Icelandic reward. On the same base model, SAGA scores higher on both parsers. Existing linguistic resources, combined with safeguards against reward hacking, can therefore stand in for human judgements. This makes grammatical alignment possible for the many languages that have parsers but no such data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 30 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

19 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization30 Jun
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  4. 2023Mitigating the Alignment Tax of RLHF12 Sep
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 14

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  7. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  9. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  10. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  11. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  12. 2022Reward Gaming in Conditional Text Generationcited by 30
2 more
  1. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  2. 2022A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Modelscited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
  2. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters
  3. 2025ComPO: Preference Alignment via Comparison Oracles
  4. 2026Meta-Learning Preferences for Multilingual LLM Alignment
  5. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
  6. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  7. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  8. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data