Does This Gradient Spark Joy?
Not on the map. Why?
Abstract (arXiv)
Policy gradient computes a backward pass for every sample, even though the backward pass is expensive and most samples carry little learning value. The Delightful Policy Gradient (DG) provides a forward-pass signal of learning value: delight, the product of advantage and surprisal (negative log-probability). We introduce the Kondo gate, which compares delight against a compute price and pays for a backward pass only when the sample is worth it, thereby tracing a quality--cost Pareto frontier. In bandits, zero-price gating preserves useful gradient signal while removing perpendicular noise, and delight is a more reliable screening signal than additive combinations of value and surprise. On MNIST and transformer token reversal, the Kondo gate skips most backward passes while retaining nearly all of DG's learning quality, with gains that grow as problems get harder and backward passes become more expensive. Because the gate tolerates approximate delight, a cheap forward pass can screen samples before expensive backpropagation, suggesting a speculative-decoding-for-training paradigm.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 11 of its 16 references that are in the map. First posted on arXiv 20 Mar 2026.
Builds on
11 of 16 references.
References · 11
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2015Trust Region Policy Optimizationcited by 2,503
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2015Prioritized Experience Replaycited by 1,070
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
- 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
- 2018Distributed Prioritized Experience Replaycited by 260
- 2017Automated Curriculum Learning for Neural Networkscited by 251
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Delightful Explorationcited by 0
- 2026Delightful Exploration13 May 2026