mlsanityData to 7 Oct 2026

Towards Spec Learning: Inference-Time Alignment from Preference Pairs

arXiv 2606.24004 ↗First posted 22 Jun 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Steering a large language model (LLM) toward a desired behavior typically relies on an iterative process of hand-crafting a prompt based on a careful inspection of the model's responses. This is an involved, brittle, and error-prone process. Preference-based fine-tuning is a more rigorous but often prohibitively expensive solution. We propose spec learning, a framework that relies on a brief user instruction and a small set of preference judgments. These are compiled into specifications in the form of natural-language prompts for an LLM. Specifications condition LLMs at inference time, and no parameter updates to the underlying models are required. We show that the responses generated based on the compiled specifications often outperform direct preference optimization (DPO) on datasets from specialized domains whose preference signal is dense. Unlike opaque weight updates, the resulting specifications are human-readable and double as interpretable and transparent written embodiments of the preference signal that produced them.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 107 of its 127 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

107 of 127 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment7 Jan
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  4. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  5. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Training7 Nov
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 99

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2024A Survey on LLM-as-a-Judgecited by 1,384
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
38 more
  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  3. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023Large Language Models as Optimizerscited by 830
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  8. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  9. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  10. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  11. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  12. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  13. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  14. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  15. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  16. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  17. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  18. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  19. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 226
  20. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  21. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  22. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  23. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  24. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  25. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  26. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  27. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  28. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  29. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  30. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  31. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  32. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  33. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  34. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
  35. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 61
  36. 2023Prompt Engineering a Prompt Engineercited by 53
  37. 2023Reprompting: Automated Chain-of-Thought Prompt Inference Through Gibbs Samplingcited by 46
  38. 2024PRewrite: Prompt Rewriting with Reinforcement Learningcited by 43

50 most cited of 99 All 99 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Inference time LLM alignment in single and multidomain preference spectrum
  2. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback
  3. 2025PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
  4. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
  5. 2025Configurable Preference Tuning with Rubric-Guided Synthetic Data
  6. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  7. 2024ICPL: Few-shot In-context Preference Learning via LLMs
  8. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier