mlsanityData to 7 Oct 2026

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time

arXiv 2410.14184 ↗First posted 18 Oct 2024cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) acquire extensive knowledge and remarkable abilities from extensive text corpora, making them powerful tools for various applications. To make LLMs more usable, aligning them with human preferences is essential. Existing alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), typically embed predefined preferences directly within the model's parameters. These methods, however, often result in a static alignment that can not account for the diversity of human preferences in practical applications. In response to this challenge, we propose an effective method, MetaAlign, which aims to help LLMs dynamically align with various explicit or implicit preferences specified at inference time. Experimental results show that LLMs optimized on our meticulously constructed MetaAlign Dataset can effectively align with any preferences specified at the inference stage, validating the feasibility of MetaAlign. We hope that our work can provide some insights into the alignment of language models.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 33 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

22 of 33 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  2. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback21 Jan
  3. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  4. 2023LIMA: Less Is More for Alignment18 May
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  9. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  10. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  11. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  12. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
4 more
  1. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  2. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  3. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  4. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024LongSafety: Enhance Safety for Long-Context LLMscited by 11
  2. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  3. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuningcited by 2
  4. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  5. 2025UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasetscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2026Meta-Learning Preferences for Multilingual LLM Alignment
  3. 2023On Diversified Preferences of Large Language Model Alignment
  4. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  5. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
  6. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  7. 2025ComPO: Preference Alignment via Comparison Oracles
  8. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey