# SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models

arXiv: https://arxiv.org/abs/2608.06179 · HTML: https://mlsanity.com/paper/2608.06179/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 19 of 19)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Mitigating the Alignment Tax of RLHF](https://mlsanity.com/paper/2309.06256/) (2023)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [A New Massive Multilingual Dataset for High-Performance Language Technologies](https://mlsanity.com/paper/2403.14009/) (2024)
- [Reward Gaming in Conditional Text Generation](https://mlsanity.com/paper/2211.08714/) (2022)
- [NorBench -- A Benchmark for Norwegian Language Models](https://mlsanity.com/paper/2305.03880/) (2023)
- [A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models](https://mlsanity.com/paper/2201.05601/) (2022)
- [BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization](https://mlsanity.com/paper/2407.00693/) (2024)

## Abstract (arXiv)

Language models for low-resource languages still make frequent grammatical errors. Alignment methods such as RLHF and DPO can correct them. However, they require human judgements, which are scarce for languages such as Icelandic. Automatic rewards offer an alternative, yet models can learn to exploit them without improving (reward hacking). We present SAGA (Score-weighted Adaptive Generation Alignment), which builds preference pairs using dependency parsers. Such parsers already exist for many low-resource languages. SAGA's reward combines parse quality with lexical diversity. It keeps only pairs whose reward gap reaches a threshold and trains on them with DPO. On Danish, Icelandic and Norwegian Bokmaal, SAGA raises reward-parser success by 18 to 28 points. It improves all four model families we test, although one Icelandic run is flagged for reward hacking. An independent parser confirms the gains, and native speakers prefer SAGA outputs in Danish (69%) and Icelandic (80%). Online RL methods exploit the Icelandic reward. On the same base model, SAGA scores higher on both parsers. Existing linguistic resources, combined with safeguards against reward hacking, can therefore stand in for human judgements. This makes grammatical alignment possible for the many languages that have parsers but no such data.