mlsanityData to 7 Oct 2026

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

arXiv 1609.08144 ↗First posted 26 Sep 2016cs.CL cs.AI cs.LGcited by 1,471

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.

Key facts. Cited by 1,471 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 2; 2018: 2; 2019: 1; 2020: 2; 2021: 452; 2022: 340; 2023: 274; 2024: 195; 2025: 129; 2026: 74.

1,471 papers.

Sort:

By area:

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  11. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  12. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
38 more
  1. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  2. 2023The Falcon Series of Open Language Modelscited by 526
  3. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  4. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  5. 2021InfographicVQAcited by 430
  6. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  8. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  9. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  10. 2021Efficient Attentions for Long Document Summarizationcited by 342
  11. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  12. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  13. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  14. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  15. 2022Transformers Learn Shortcuts to Automatacited by 260
  16. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  17. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  18. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  19. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  20. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  21. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  22. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  23. 2021Accelerating Sparse Deep Neural Networkscited by 213
  24. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  25. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  26. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  27. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  28. 2023Recommender Systems with Generative Retrievalcited by 191
  29. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  30. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  31. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  32. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  33. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 177
  34. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  35. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  36. 2023Conformal Language Modelingcited by 163
  37. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  38. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMscited by 162

50 most cited of 1,471 All 1,471 (JSON)