# Feedback Descent: Open-Ended Text Optimization via Pairwise Comparison

arXiv: https://arxiv.org/abs/2511.07919 · HTML: https://mlsanity.com/paper/2511.07919/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 73 of its 92 references that are in the map.
- First posted on arXiv 11 Nov 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 73)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)

## Built on by (most cited first, 30 of 44)

- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026)
- [EvoSkill: Automated Skill Discovery for Multi-Agent Systems](https://mlsanity.com/paper/2603.02766/) (2026)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolution](https://mlsanity.com/paper/2607.05297/) (2026)
- [InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning](https://mlsanity.com/paper/2601.14209/) (2026)
- [VQQA: An Agentic Approach for Video Evaluation and Quality Improvement](https://mlsanity.com/paper/2603.12310/) (2026)
- [Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learning](https://mlsanity.com/paper/2512.05105/) (2025)
- [Learning from Language Feedback via Variational Policy Distillation](https://mlsanity.com/paper/2605.15113/) (2026)
- [EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning](https://mlsanity.com/paper/2603.21728/) (2026)
- [JTPRO: A Joint Tool-Prompt Reflective Optimization Framework for Language Agents](https://mlsanity.com/paper/2604.19821/) (2026)
- [LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks](https://mlsanity.com/paper/2607.18110/) (2026)
- [Automatic Prompt Optimization for Dataset-Level Feature Discovery](https://mlsanity.com/paper/2601.13922/) (2026)
- [Why Human Guidance Matters in Collaborative Vibe Coding](https://mlsanity.com/paper/2602.10473/) (2026)
- [SeaEvo: Advancing Algorithm Discovery with Strategy Space Evolution](https://mlsanity.com/paper/2604.24372/) (2026)
- [Co-Director: Agentic Generative Video Storytelling](https://mlsanity.com/paper/2604.24842/) (2026)
- [SPIRAL: Learning to Search and Aggregate](https://mlsanity.com/paper/2606.23595/) (2026)
- [Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferences](https://mlsanity.com/paper/2602.21585/) (2026)
- [PrefPO: Pairwise Preference Prompt Optimization](https://mlsanity.com/paper/2603.19311/) (2026)
- [OpenProver: Agentic and Interactive Theorem Proving with Lean 4](https://mlsanity.com/paper/2607.09217/) (2026)
- [Visual Persuasion: What Influences Decisions of Vision-Language Models?](https://mlsanity.com/paper/2602.15278/) (2026)
- [Asking the Right Questions: Improving Reasoning with Generated Stepping Stones](https://mlsanity.com/paper/2602.19069/) (2026)
- [Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation](https://mlsanity.com/paper/2603.01375/) (2026)
- [CASCADE: Case-Based Continual Adaptation for Large Language Models During Deployment](https://mlsanity.com/paper/2605.06702/) (2026)
- [RIZZ: Routing Interactions to Near Zero-Interference Zones for Continual Adaptation of Black-Box Agents](https://mlsanity.com/paper/2606.20638/) (2026)
- [Building Agent Harnesses for Scientific Curation from Multimodal Sources](https://mlsanity.com/paper/2606.21005/) (2026)
- [Freeform Preference Learning for Robotic Manipulation](https://mlsanity.com/paper/2606.32027/) (2026)
- [Heuristic Learning for Active Flow Control Using Coding Agents](https://mlsanity.com/paper/2607.11565/) (2026)
- [MEGA: Self-Evolving Agent Optimization Infrastructure via Wisdom Graph](https://mlsanity.com/paper/2608.10504/) (2026)
- [Improving Information Extraction with Learned Queries](https://mlsanity.com/paper/2608.31058/) (2026)

## Abstract (arXiv)

We introduce textitFeedback Descent, a framework that optimizes text artifacts -- prompts, code, and molecules -- through structured textual feedback, rather than relying solely on scalar rewards. By preserving detailed critiques instead of compressing them to binary preferences, Feedback Descent widens the information bottleneck in preference learning, enabling directed optimization in text space rather than weight space. We show that in-context learning can transform structured feedback into gradient-like directional information, enabling targeted edits. Unlike prior approaches that collapse judgments into single bits, our evaluators pair each comparison with textual feedback, which functions as high-bandwidth supervision. The iteration loop is done purely at inference time, without modifying any model weights, and is task-agnostic. We evaluate Feedback Descent on three diverse domains and find that it outperforms state-of-the-art prompt optimization (GEPA), reinforcement learning methods (GRPO, REINVENT), and even specialized graph-based molecular optimizers. In the DOCKSTRING molecule discovery benchmark, Feedback Descent identifies novel drug-like molecules surpassing the 99.9th percentile of a database with more than 260,000 compounds across six protein targets.