mlsanityData to 14 Oct 2026

Lagrangian Method for Q-Function Learning (with Applications to Machine Translation)

arXiv 2207.11161 ↗First posted 22 Jul 2022cs.LG cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper discusses a new approach to the fundamental problem of learning optimal Q-functions. In this approach, optimal Q-functions are formulated as saddle points of a nonlinear Lagrangian function derived from the classic Bellman optimality equation. The paper shows that the Lagrangian enjoys strong duality, in spite of its nonlinearity, which paves the way to a general Lagrangian method to Q-function learning. As a demonstration, the paper develops an imitation learning algorithm based on the duality theory, and applies the algorithm to a state-of-the-art machine translation benchmark. The paper then turns to demonstrate a symmetry breaking phenomenon regarding the optimality of the Lagrangian saddle points, which justifies a largely overlooked direction in developing the Lagrangian method.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 22 Jul 2022.

Builds on

30 of 52 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2021Convex optimizationcited by 2,515
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  8. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  9. 2017Six Challenges for Neural Machine Translationcited by 335
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  12. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 159
18 more
  1. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  2. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  3. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  4. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  5. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  6. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  7. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  8. 2017Classical Structured Prediction Losses for Sequence to Sequence Learningcited by 55
  9. 2016Unifying task specification in reinforcement learningcited by 41
  10. 2020Steady State Analysis of Episodic Reinforcement Learningcited by 21
  11. 2016Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learningcited by 15
  12. 2017Boosting the Actor with Dual Criticcited by 14
  13. 2017Randomized Linear Programming Solves the Discounted Markov Decision Problem In Nearly-Linear (Sometimes Sublinear) Running Timecited by 11
  14. 2018Scalable Bilinear π Learning Using State and Action Featurescited by 10
  15. 2019Faster saddle-point optimization for solving large-scale Markov decision processescited by 8
  16. 2018Stochastic Primal-Dual Q-Learningcited by 7
  17. 2017Deep Primal-Dual Reinforcement Learning: Accelerating Actor-Critic using Bellman Dualitycited by 3
  18. 2021Simpson's Bias in NLP Trainingcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Utility-Probability Duality of Neural Networkscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Q-Learning with Stable Infinite-Dimensional Linear Function Approximation
  2. 2026Optimistic Training and Convergence of Q-Learning -- Extended Version
  3. 2024Gaussian-Mixture-Model Q-Functions for Reinforcement Learning by Riemannian Optimization
  4. 2025Physics of Learning: A Lagrangian perspective to different learning paradigms
  5. 2025From Set Convergence to Pointwise Convergence: Finite-Time Guarantees for Average-Reward Q-Learning with Adaptive Stepsizes
  6. 2025Alignment of large language models with constrained learning
  7. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning
  8. 2025Linear Q-Learning Does Not Diverge in L2: Convergence Rates to a Bounded Set