mlsanityData to 14 Oct 2026

Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

arXiv 2609.22226 ↗First posted 3 Sep 2026cs.CL cs.LG stat.MLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Decode-time alignment methods steer a frozen language model by scoring candidate continuations with an external reward and selecting the maximiser. We argue that this shared design is a single degenerate point in a much larger space. We introduce Swiss-Knife, a framework for externalised multi-objective alignment in which the alignment specification is a first-class runtime object: hot-swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule. Six axioms characterise the admissible aggregation operators, and we prove a representation theorem: every operator satisfying them has the form Ri = sumj neq i g((μi - μj)/s(σi,σj)), a two-parameter family containing probit and logistic comparison rules and pointwise argmax as named coordinates. Within it, pairwise aggregation is Lipschitz-stable under adversarial reward contamination while argmax is not, and Candidate-Batch Normalization (CBN) makes the weight simplex invariant to the rescalings under which reward models are only ever identified. Our reference instantiation pairs DPO-LoRA blades with an uncertainty-aware pairwise tournament. Sweeping the helpfulness/honesty/harmlessness simplex, it attains the best balanced frontier of six decode-time methods (harmonic F1 0.797 vs. 0.750 for the strongest baseline, p < 10-14) with the lowest refusal rate and highest helpfulness of any arm, and reconfigures its objectives in 0.05 ms with no gradient computation. Ablating CBN costs 0.217 F1, and the metadata confirms the predicted mechanism: the lowest-variance blade retains 9% of its nominal 33% influence, and the collapse follows the predicted ordering.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 29 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

13 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  2. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignment24 Jun
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 10

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  8. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  10. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Prompts
  2. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
  3. 2026MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models
  4. 2026Reward-free Alignment for Conflicting Objectives
  5. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignment
  6. 2025Pareto Multi-Objective Alignment for Language Models
  7. 2026MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment
  8. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selection