# Bandit Structured Prediction for Neural Sequence-to-Sequence Learning

arXiv: https://arxiv.org/abs/1704.06497 · HTML: https://mlsanity.com/paper/1704.06497/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2017.

## Built on by (most cited first, 30 of 31)

- [ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models](https://mlsanity.com/paper/2310.10505/) (2023)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback](https://mlsanity.com/paper/1707.07402/) (2017)
- [DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO](https://mlsanity.com/paper/2506.07464/) (2025)
- [Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning](https://mlsanity.com/paper/1805.10627/) (2018)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/2106.08942/) (2021)
- [Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model](https://mlsanity.com/paper/2401.12873/) (2024)
- [Interactive Learning from Activity Description](https://mlsanity.com/paper/2102.07024/) (2021)
- [Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior](https://mlsanity.com/paper/2108.04812/) (2021)
- [Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits](https://mlsanity.com/paper/2110.06997/) (2021)
- [ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation](https://mlsanity.com/paper/2308.02223/) (2023)
- [Teacher Forcing Recovers Reward Functions for Text Generation](https://mlsanity.com/paper/2210.08708/) (2022)
- [Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings](https://mlsanity.com/paper/2411.05986/) (2024)
- [Aligning to What? Limits to RLHF Based Alignment](https://mlsanity.com/paper/2503.09025/) (2025)
- [Stochastic Multi-Armed Bandits with Control Variates](https://mlsanity.com/paper/2105.03962/) (2021)
- [MAD for Robust Reinforcement Learning in Machine Translation](https://mlsanity.com/paper/2207.08583/) (2022)
- [Exploiting Correlated Auxiliary Feedback in Parameterized Bandits](https://mlsanity.com/paper/2311.02715/) (2023)
- [LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectives](https://mlsanity.com/paper/2407.01490/) (2024)
- [SumRecom: A Personalized Summarization Approach by Learning from Users' Feedback](https://mlsanity.com/paper/2408.07294/) (2024)
- [Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security](https://mlsanity.com/paper/2507.22037/) (2025)
- [Fixing exposure bias with imitation learning needs powerful oracles](https://mlsanity.com/paper/2109.04114/) (2021)
- [SPEECH: Structured Prediction with Energy-Based Event-Centric Hyperspheres](https://mlsanity.com/paper/2305.13617/) (2023)
- [One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF](https://mlsanity.com/paper/2503.19523/) (2025)
- [Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewards](https://mlsanity.com/paper/2601.04411/) (2026)
- [Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback](https://mlsanity.com/paper/2606.01081/) (2026)
- [Stochastic Multi-Armed Bandits with Limited Control Variates](https://mlsanity.com/paper/2603.02100/) (2026)
- [Reference-Free Reinforcement Learning Fine-Tuning for MT: A Seq2Seq Perspective](https://mlsanity.com/paper/2605.15976/) (2026)
- [Bridging the Usability Gap: Lessons from Interpreting Studies for Machine Interpreting Design](https://mlsanity.com/paper/2606.16009/) (2026)

## Abstract (arXiv)

Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures. We advance this framework by lifting linear bandit learning to neural sequence-to-sequence learning problems using attention-based recurrent neural networks. Furthermore, we show how to incorporate control variates into our learning algorithms for variance reduction and improved generalization. We present an evaluation on a neural machine translation task that shows improvements of up to 5.89 BLEU points for domain adaptation from simulated bandit feedback.