mlsanityData to 14 Oct 2026

Feedback Descent: Open-Ended Text Optimization via Pairwise Comparison

arXiv 2511.07919 ↗First posted 11 Nov 2025cs.LGcited by 44

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

We introduce textitFeedback Descent, a framework that optimizes text artifacts -- prompts, code, and molecules -- through structured textual feedback, rather than relying solely on scalar rewards. By preserving detailed critiques instead of compressing them to binary preferences, Feedback Descent widens the information bottleneck in preference learning, enabling directed optimization in text space rather than weight space. We show that in-context learning can transform structured feedback into gradient-like directional information, enabling targeted edits. Unlike prior approaches that collapse judgments into single bits, our evaluators pair each comparison with textual feedback, which functions as high-bandwidth supervision. The iteration loop is done purely at inference time, without modifying any model weights, and is task-agnostic. We evaluate Feedback Descent on three diverse domains and find that it outperforms state-of-the-art prompt optimization (GEPA), reinforcement learning methods (GRPO, REINVENT), and even specialized graph-based molecular optimizers. In the DOCKSTRING molecule discovery benchmark, Feedback Descent identifies novel drug-like molecules surpassing the 99.9th percentile of a database with more than 260,000 compounds across six protein targets.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 73 of its 92 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

73 of 92 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Training Language Models with Language Feedback at Scale28 Mar
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 71

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
38 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,637
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  11. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  13. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  14. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  15. 2023Large Language Models as Optimizerscited by 864
  16. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  17. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  18. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  19. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  20. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
  21. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  22. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  23. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  24. 2022Self-critiquing models for assisting human evaluatorscited by 380
  25. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  26. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  27. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  28. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  29. 2023Tool Learning with Foundation Modelscited by 302
  30. 2018GuacaMol: Benchmarking Models for De Novo Molecular Designcited by 297
  31. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
  32. 2022Generating Sequences by Learning to Self-Correctcited by 277
  33. 2024TextGrad: Automatic "Differentiation" via Textcited by 273
  34. 2017Molecular De Novo Design through Deep Reinforcement Learningcited by 267
  35. 2023Nash Learning from Human Feedbackcited by 224
  36. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
  37. 2022TALM: Tool Augmented Language Modelscited by 197
  38. 2025Generalizing Verifiable Instruction Followingcited by 194

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 43.

44 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 309
  3. 2026EvoSkill: Automated Skill Discovery for Multi-Agent Systemscited by 107
  4. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  5. 2026MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolutioncited by 22
  6. 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoningcited by 14
  7. 2026VQQA: An Agentic Approach for Video Evaluation and Quality Improvementcited by 12
  8. 2025Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learningcited by 9
  9. 2026Learning from Language Feedback via Variational Policy Distillationcited by 7
  10. 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learningcited by 6
  11. 2026JTPRO: A Joint Tool-Prompt Reflective Optimization Framework for Language Agentscited by 4
  12. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 4
32 more
  1. 2026Automatic Prompt Optimization for Dataset-Level Feature Discoverycited by 3
  2. 2026Why Human Guidance Matters in Collaborative Vibe Codingcited by 3
  3. 2026SeaEvo: Advancing Algorithm Discovery with Strategy Space Evolutioncited by 3
  4. 2026Co-Director: Agentic Generative Video Storytellingcited by 3
  5. 2026SPIRAL: Learning to Search and Aggregatecited by 3
  6. 2026Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferencescited by 2
  7. 2026PrefPO: Pairwise Preference Prompt Optimizationcited by 2
  8. 2026OpenProver: Agentic and Interactive Theorem Proving with Lean 4cited by 2
  9. 2026Visual Persuasion: What Influences Decisions of Vision-Language Models?cited by 1
  10. 2026Asking the Right Questions: Improving Reasoning with Generated Stepping Stonescited by 1
  11. 2026Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptationcited by 1
  12. 2026CASCADE: Case-Based Continual Adaptation for Large Language Models During Deploymentcited by 1
  13. 2026RIZZ: Routing Interactions to Near Zero-Interference Zones for Continual Adaptation of Black-Box Agentscited by 1
  14. 2026Building Agent Harnesses for Scientific Curation from Multimodal Sourcescited by 1
  15. 2026Freeform Preference Learning for Robotic Manipulationcited by 1
  16. 2026Heuristic Learning for Active Flow Control Using Coding Agentscited by 1
  17. 2026MEGA: Self-Evolving Agent Optimization Infrastructure via Wisdom Graphcited by 1
  18. 2026Improving Information Extraction with Learned Queriescited by 1
  19. 2026Optimizing Multi-Agent Weather Captioning via Text Gradient Descent: A Training-Free Approach with Consensus-Aware Gradient Fusioncited by 0
  20. 2026Efficiently Aligning Language Models with Online Natural Language Feedbackcited by 0
  21. 2026Prompt Codebooks: Discrete Compositional Optimization for Language Model Instruction Refinementcited by 0
  22. 2026Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimizationcited by 0
  23. 2026When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?cited by 0
  24. 2026MUSE: A Unified Agentic Harness for MLLMscited by 0
  25. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
  26. 2026Reward-Free Evolving Agents via Pairwise Validatorcited by 0
  27. 2026From Agent Failures to Text Policies: What Works and What Breakscited by 0
  28. 2026Evaluation-as-Search: Adaptive Discovery of Grounding Failures in Meeting Assistantscited by 0
  29. 2026Learning to Coach for Experiential Learningcited by 0
  30. 2026Training LLM Judges from Language Feedback via Position-Selective Self-Distillationcited by 0
  31. 2026GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolutioncited by 0
  32. 2026SHELDON: Reasoning through Dual Search over Hypothesis and Observation Spacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Text2Grad: Reinforcement Learning from Natural Language Feedback
  2. 2026STEVE: Stabilizing Textual Gradient-Based Prompt Optimization via Error-Driven Refinement and Regularized Verification
  3. 2025MAPGD: Multi-Agent Prompt Gradient Descent for Collaborative Prompt Optimization
  4. 2025Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans
  5. 2026MFG: Multi-turn Feedback Guided Molecular Optimization
  6. 2025LILO: Bayesian Optimization with Natural Language Feedback
  7. 2026TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuning
  8. 2026Scaffold-Conditioned Preference Triplets for Controllable Molecular Optimization with Large Language Models