mlsanityData to 14 Oct 2026

Classical Structured Prediction Losses for Sequence to Sequence Learning

arXiv 1711.04956 ↗First posted 14 Nov 2017cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam. In this paper, we survey a range of classical objective functions that have been widely used to train linear models for structured prediction and apply them to neural sequence to sequence models. Our experiments show that these losses can perform surprisingly well by slightly outperforming beam search optimization in a like for like setup. We also report new state of the art results on both IWSLT'14 German-English translation as well as Gigaword abstractive summarization. On the larger WMT'14 English-French translation task, sequence-level training achieves 41.5 BLEU which is on par with the state of the art.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 19; 2023: 11; 2024: 4; 2025: 1; 2026: 2.

55 papers.

Sort:

By area:

  1. 2021Random Feature Attentioncited by 354
  2. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  3. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  4. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  5. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  6. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  7. 2021Survey of Low-Resource Machine Translationcited by 92
  8. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  9. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  10. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  11. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  12. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
38 more
  1. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 49
  2. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  3. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  4. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 33
  5. 2021Bayesian Attention Belief Networkscited by 25
  6. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  7. 2024Understanding Reference Policies in Direct Preference Optimizationcited by 22
  8. 2021BERTTune: Fine-Tuning Neural Machine Translation with BERTScorecited by 18
  9. 2021Sequence-Level Training for Non-Autoregressive Neural Machine Translationcited by 18
  10. 2021Modelling Latent Translations for Cross-Lingual Transfercited by 16
  11. 2023ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generationcited by 15
  12. 2022Hierarchical classification at multiple operating pointscited by 14
  13. 2021Reward Optimization for Neural Machine Translation with Learned Metricscited by 12
  14. 2022A Call for Clarity in Beam Search: How It Works and When It Stopscited by 10
  15. 2023Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsetscited by 10
  16. 2021Mixed Cross Entropy Loss for Neural Machine Translationcited by 8
  17. 2022Momentum Calibration for Text Generationcited by 8
  18. 2022MAD for Robust Reinforcement Learning in Machine Translationcited by 7
  19. 2022Multi-Granularity Optimization for Non-Autoregressive Translationcited by 6
  20. 2021RewardsOfSum: Exploring Reinforcement Learning Rewards for Summarisationcited by 4
  21. 2022AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translationcited by 4
  22. 2022Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learningcited by 4
  23. 2023Beyond MLE: Convex Learning for Text Generationcited by 4
  24. 2025RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translationcited by 4
  25. 2021Simpson's Bias in NLP Trainingcited by 3
  26. 2021What's Hidden in a One-layer Randomly Weighted Transformer?cited by 3
  27. 2022Focus on the Target's Vocabulary: Masked Label Smoothing for Machine Translationcited by 3
  28. 2023Tuna: Instruction Tuning using Feedback from Large Language Modelscited by 3
  29. 2026Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Modelscited by 3
  30. 2022Are Neighbors Enough? Multi-Head Neural n-gram can be Alternative to Self-attentioncited by 2
  31. 2022Reinforcement Learning with Large Action Spaces for Neural Machine Translationcited by 2
  32. 2023On the Learning Dynamics of Attention Networkscited by 2
  33. 2022Lagrangian Method for Q-Function Learning (with Applications to Machine Translation)cited by 1
  34. 2022GROOT: Corrective Reward Optimization for Generative Sequential Labelingcited by 1
  35. 2023Finding the Pillars of Strength for Multi-Head Attentioncited by 1
  36. 2024Diverse Sign Language Translationcited by 1
  37. 2026DIFF-ERO: A Conformance-Aware Loss for Deep Learning in Process Miningcited by 1
  38. 2021Cascaded Head-colliding Attentioncited by 0

50 most cited of 55 All 55 (JSON)