# ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation

arXiv: https://arxiv.org/abs/2005.00850 · HTML: https://mlsanity.com/paper/2005.00850/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 30 of 34)

- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023)
- [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://mlsanity.com/paper/2206.00761/) (2022)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application](https://mlsanity.com/paper/2407.01885/) (2024)
- [Token Dropping for Efficient BERT Pretraining](https://mlsanity.com/paper/2203.13240/) (2022)
- [Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings](https://mlsanity.com/paper/2106.02736/) (2021)
- [Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models](https://mlsanity.com/paper/2412.14528/) (2024)
- [Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretations](https://mlsanity.com/paper/2401.14142/) (2024)
- [BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation](https://mlsanity.com/paper/2406.13555/) (2024)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [Sinkhorn Distance Minimization for Knowledge Distillation](https://mlsanity.com/paper/2402.17110/) (2024)
- [Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models](https://mlsanity.com/paper/2101.06829/) (2021)
- [Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translation](https://mlsanity.com/paper/2301.11503/) (2023)
- [Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference](https://mlsanity.com/paper/2412.18934/) (2024)
- [Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2303.06662/) (2023)
- [Energy-Based Models for Code Generation under Compilability Constraints](https://mlsanity.com/paper/2106.04985/) (2021)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfer](https://mlsanity.com/paper/2106.02210/) (2021)
- [Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End](https://mlsanity.com/paper/2212.10522/) (2022)
- [Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn't Matter (Much)](https://mlsanity.com/paper/2502.04499/) (2025)
- [Amortized Noisy Channel Neural Machine Translation](https://mlsanity.com/paper/2112.08670/) (2021)
- [Rephrasing the Reference for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2211.16863/) (2022)
- [Energy-Based Models for Cross-Modal Localization using Convolutional Transformers](https://mlsanity.com/paper/2306.04021/) (2023)
- [LLMR: Knowledge Distillation with a Large Language Model-Induced Reward](https://mlsanity.com/paper/2409.12500/) (2024)
- [Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge](https://mlsanity.com/paper/2604.02621/) (2026)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)
- [Aligning language models with human preferences](https://mlsanity.com/paper/2404.12150/) (2024)
- [DiMS: Distilling Multiple Steps of Iterative Non-Autoregressive Transformers for Machine Translation](https://mlsanity.com/paper/2206.02999/) (2022)
- [EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering](https://mlsanity.com/paper/2206.14355/) (2022)

## Abstract (arXiv)

We propose to train a non-autoregressive machine translation model to minimize the energy defined by a pretrained autoregressive model. In particular, we view our non-autoregressive translation system as an inference network (Tu and Gimpel, 2018) trained to minimize the autoregressive teacher energy. This contrasts with the popular approach of training a non-autoregressive model on a distilled corpus consisting of the beam-searched outputs of such a teacher model. Our approach, which we call ENGINE (ENerGy-based Inference NEtworks), achieves state-of-the-art non-autoregressive results on the IWSLT 2014 DE-EN and WMT 2016 RO-EN datasets, approaching the performance of autoregressive models.