mlsanityData to 7 Oct 2026

ZEBRA: Leveraging Model-Behavioral Knowledge for Zero-Annotation Preference Dataset Construction

arXiv 2502.18744 ↗First posted 26 Feb 2025cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent efforts in LLM alignment have focused on constructing large-scale preference datasets via human or Artificial Intelligence (AI) annotators. However, such approaches rely on instance-wise supervision, incurring substantial annotation cost and limited interpretability. In this paper, we propose ZEBRA - a model behavior-wise zero-annotation framework that constructs preference data by leveraging model behavior knowledge derived from benchmark performances. ZEBRA binarizes response pairs by evaluating the quality and similarity of their origin models, entirely bypassing instance-level annotation. This allows scalable, controllable, and cost-effective alignment data generation. Empirical results show that ZEBRA achieves alignment performance comparable to instance-supervised methods, despite requiring no manual or model-based labeling.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 32 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

18 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
4 more
  1. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  2. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  3. 2023Safer-Instruct: Aligning Language Models with Automated Preference Datacited by 27
  4. 2024SAGE-RT: Synthetic Alignment data Generation for Safety Evaluation and Red Teamingcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  2. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset
  3. 2024Annotation-Efficient Language Model Alignment via Diverse and Representative Response Texts
  4. 2025Alignment Data Map for Efficient Preference Data Selection and Diagnosis
  5. 2025HEAL: A Hypothesis-Based Preference-Aware Analysis Framework
  6. 2025How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators
  7. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  8. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment