mlsanityData to 7 Oct 2026

ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation

arXiv 2005.00850 ↗First posted 2 May 2020cs.CL cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

We propose to train a non-autoregressive machine translation model to minimize the energy defined by a pretrained autoregressive model. In particular, we view our non-autoregressive translation system as an inference network (Tu and Gimpel, 2018) trained to minimize the autoregressive teacher energy. This contrasts with the popular approach of training a non-autoregressive model on a distilled corpus consisting of the beam-searched outputs of such a teacher model. Our approach, which we call ENGINE (ENerGy-based Inference NEtworks), achieves state-of-the-art non-autoregressive results on the IWSLT 2014 DE-EN and WMT 2016 RO-EN datasets, approaching the performance of autoregressive models.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 4; 2024: 9; 2025: 1; 2026: 3.

34 papers.

Sort:

By area:

  1. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  2. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  3. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  4. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  5. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  6. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  7. 2021Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastingscited by 37
  8. 2024Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Modelscited by 35
  9. 2024Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretationscited by 33
  10. 2021Sequence-Level Training for Non-Autoregressive Neural Machine Translationcited by 18
  11. 2024BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillationcited by 16
  12. 2024Sinkhorn Distance Minimization for Knowledge Distillationcited by 15
22 more
  1. 2021Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Modelscited by 14
  2. 2023Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translationcited by 13
  3. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  4. 2023Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translationcited by 11
  5. 2021Energy-Based Models for Code Generation under Compilability Constraintscited by 10
  6. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  7. 2021NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfercited by 7
  8. 2022Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-Endcited by 6
  9. 2025Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn't Matter (Much)cited by 5
  10. 2021Amortized Noisy Channel Neural Machine Translationcited by 4
  11. 2022Rephrasing the Reference for Non-Autoregressive Machine Translationcited by 3
  12. 2023Energy-Based Models for Cross-Modal Localization using Convolutional Transformerscited by 3
  13. 2024LLMR: Knowledge Distillation with a Large Language Model-Induced Rewardcited by 3
  14. 2026Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judgecited by 2
  15. 2021Goal-driven text descriptions for imagescited by 1
  16. 2024Aligning language models with human preferencescited by 1
  17. 2022DiMS: Distilling Multiple Steps of Iterative Non-Autoregressive Transformers for Machine Translationcited by 0
  18. 2022EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answeringcited by 0
  19. 2022Implicit Training of Energy Model for Structure Predictioncited by 0
  20. 2024QE-EBM: Using Quality Estimators as Energy Loss for Machine Translationcited by 0
  21. 2026NADIR: Differential Attention Flow for Non-Autoregressive Transliteration in Indic Languagescited by 0
  22. 2026Interpreting Black-Box Large Language Models with Sentence-Level Energy Landscapescited by 0