mlsanityData to 7 Oct 2026

Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles

arXiv 2303.03751 ↗First posted 7 Mar 2023cs.LG cs.AIcited by 34

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In this study, we delve into an emerging optimization challenge involving a black-box objective function that can only be gauged via a ranking oracle-a situation frequently encountered in real-world scenarios, especially when the function is evaluated by human judges. Such challenge is inspired from Reinforcement Learning with Human Feedback (RLHF), an approach recently employed to enhance the performance of Large Language Models (LLMs) using human guidance. We introduce ZO-RankSGD, an innovative zeroth-order optimization algorithm designed to tackle this optimization problem, accompanied by theoretical assurances. Our algorithm utilizes a novel rank-based random estimator to determine the descent direction and guarantees convergence to a stationary point. Moreover, ZO-RankSGD is readily applicable to policy optimization problems in Reinforcement Learning (RL), particularly when only ranking oracles for the episode reward are available. Last but not least, we demonstrate the effectiveness of ZO-RankSGD in a novel application: improving the quality of images generated by a diffusion generative model with human ranking feedback. Throughout experiments, we found that ZO-RankSGD can significantly enhance the detail of generated images with only a few rounds of human feedback. Overall, our work advances the field of zeroth-order optimization by addressing the problem of optimizing functions with only ranking feedback, and offers a new and effective approach for aligning Artificial Intelligence (AI) with human intentions.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 80 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

30 of 80 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  10. 2018A Tutorial on Bayesian Optimizationcited by 734
  11. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  12. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
17 more
  1. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  2. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  3. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
  4. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  5. 2020Locally Differentially Private (Contextual) Bandits Learningcited by 52
  6. 2016CMA-ES for Hyperparameter Optimization of Deep Neural Networkscited by 47
  7. 2019Gradientless Descent: High-Dimensional Zeroth-Order Optimizationcited by 42
  8. 2022Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomescited by 41
  9. 2021Differentially Private Federated Bayesian Optimization with Distributed Explorationcited by 32
  10. 2018SEGA: Variance Reduction via Gradient Sketchingcited by 29
  11. 2017Regret Analysis for Continuous Dueling Banditcited by 21
  12. 2019Multi-Attribute Bayesian Optimization With Interactive Preference Learningcited by 21
  13. 2020Local Differential Privacy for Bayesian Optimizationcited by 19
  14. 2023z-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learningcited by 17
  15. 2021Oneshot Differentially Private Top-k Selectioncited by 16
  16. 2020A One-bit, Comparison-Based Gradient Estimatorcited by 15
  17. 2021Low-rank Matrix Recovery With Unknown Correspondencecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 15; 2025: 6; 2026: 7.

34 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  3. 2024Accelerating Parallel Sampling of Diffusion Modelscited by 30
  4. 2024A New Creative Generation Pipeline for Click-Through Rate with Stable Diffusion Modelcited by 26
  5. 2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
  6. 2023FABRIC: Personalizing Diffusion Models with Iterative Feedbackcited by 19
  7. 2024ViPer: Visual Personalization of Generative Models via Individual Preference Learningcited by 17
  8. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
  9. 2024Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generationcited by 15
  10. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  11. 2025ComPO: Preference Alignment via Comparison Oraclescited by 11
  12. 2024Preference Adaptive and Sequential Text-to-Image Generationcited by 8
22 more
  1. 2023Toward 6G Native-AI Network: Foundation Model based Cloud-Edge-End Collaboration Frameworkcited by 7
  2. 2023HuTuMotion: Human-Tuned Navigation of Latent Motion Diffusion Models with Minimal Feedbackcited by 6
  3. 2024CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representationscited by 5
  4. 2025ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimizationcited by 4
  5. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
  6. 2025Instant Preference Alignment for Text-to-Image Diffusion Modelscited by 4
  7. 2023Optimizing Algorithms From Pairwise User Preferencescited by 3
  8. 2024CoCoG-2: Controllable generation of visual stimuli for understanding human concept representationcited by 3
  9. 2024Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidancecited by 3
  10. 2026Oracle Noise: Faster Semantic Spherical Alignment for Interpretable Latent Optimizationcited by 2
  11. 2026AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Momentscited by 2
  12. 2024Human Aesthetic Preference-Based Large Text-to-Image Model Personalization: Kandinsky Generation as an Examplecited by 1
  13. 2024Deep Representation Learning for Multi-functional Degradation Modeling of Community-dwelling Aging Populationcited by 1
  14. 2026Two-Fidelity Best-Action Identification for Stochastic Minimax Treecited by 1
  15. 2024Gradient Testing and Estimation by Comparisonscited by 0
  16. 2024Ruppert-Polyak averaging for Stochastic Order Oraclecited by 0
  17. 2025Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-Task Offline RLcited by 0
  18. 2025Explicit and Non-asymptotic Query Complexities of Rank-Based Zeroth-order Algorithm on Stochastic Smooth Functionscited by 0
  19. 2026Riemannian Dueling Optimizationcited by 0
  20. 2026Finding Stationary Points by Comparisonscited by 0
  21. 2026Sculpting NeRF Geometry: Human-Preference Fine-Tuning of a 3D-Aware Face GANcited by 0
  22. 2026A Zeroth-Order Paradigm for LLM Preference Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024The Perfect Blend: Redefining RLHF with Mixture of Judges
  5. 2024David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training
  6. 2025Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback
  7. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  8. 2025Reinforcement Learning from Human Feedback