mlsanityData to 7 Oct 2026

An empirical analysis of phrase-based and neural machine translation

arXiv 2103.03108 ↗First posted 4 Mar 2021cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Two popular types of machine translation (MT) are phrase-based and neural machine translation systems. Both of these types of systems are composed of multiple complex models or layers. Each of these models and layers learns different linguistic aspects of the source language. However, for some of these models and layers, it is not clear which linguistic phenomena are learned or how this information is learned. For phrase-based MT systems, it is often clear what information is learned by each model, and the question is rather how this information is learned, especially for its phrase reordering model. For neural machine translation systems, the situation is even more complex, since for many cases it is not exactly clear what information is learned and how it is learned. To shed light on what linguistic phenomena are captured by MT systems, we analyze the behavior of important models in both phrase-based and neural MT systems. We consider phrase reordering models from phrase-based MT systems to investigate which words from inside of a phrase have the biggest impact on defining the phrase reordering behavior. Additionally, to contribute to the interpretability of neural MT systems we study the behavior of the attention model, which is a key component in neural MT systems and the closest model in functionality to phrase reordering models in phrase-based systems. The attention model together with the encoder hidden state representations form the main components to encode source side linguistic information in neural MT. To this end, we also analyze the information captured in the encoder hidden state representations of a neural MT system. We investigate the extent to which syntactic and lexical-semantic information from the source side is captured by hidden state representations of different neural MT architectures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 54 of its 142 references that are in the map. First posted on arXiv 4 Mar 2021.

Builds on

54 of 142 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  11. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
38 more
  1. 2017Convolutional Sequence to Sequence Learningcited by 550
  2. 2016Neural Architectures for Named Entity Recognitioncited by 548
  3. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  4. 2017Six Challenges for Neural Machine Translationcited by 334
  5. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  6. 2018Scaling Neural Machine Translationcited by 204
  7. 2018Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learningcited by 203
  8. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  9. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  10. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  11. 2016Modeling Coverage for Neural Machine Translationcited by 110
  12. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  13. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  14. 2017Recent Advances in Recurrent Neural Networkscited by 89
  15. 2014On Using Very Large Target Vocabulary for Neural Machine Translationcited by 80
  16. 2018Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Taskscited by 56
  17. 2017Massive Exploration of Neural Machine Translation Architecturescited by 55
  18. 2014Addressing the Rare Word Problem in Neural Machine Translationcited by 50
  19. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  20. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  21. 2016Representation of linguistic form and function in recurrent neural networkscited by 39
  22. 2016Linguistic Input Features Improve Neural Machine Translationcited by 38
  23. 2016Neural Machine Translation with Supervised Attentioncited by 31
  24. 2017What does Attention in Neural Machine Translation Pay Attention to?cited by 30
  25. 2018Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architecturescited by 30
  26. 2016Problems With Evaluation of Word Embeddings Using Word Similarity Taskscited by 25
  27. 2018A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translationcited by 23
  28. 2016Neural versus Phrase-Based Machine Translation Quality: a Case Studycited by 19
  29. 2018Interpreting Neural Networks With Nearest Neighborscited by 18
  30. 2016Incorporating Structural Alignment Biases into an Attentional Neural Translation Modelcited by 16
  31. 2018An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translationcited by 15
  32. 2017Making Sense of Word Embeddingscited by 12
  33. 2016Guided Alignment Training for Topic-Aware Neural Machine Translationcited by 11
  34. 2018Learning When to Concentrate or Divert Attention: Self-Adaptive Attention Temperature for Neural Machine Translationcited by 10
  35. 2016Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translationcited by 8
  36. 2016Supervised Attentions for Neural Machine Translationcited by 8
  37. 2019Interrogating the Explanatory Power of Attention in Neural Machine Translationcited by 7
  38. 2018An Operation Sequence Model for Explainable Neural Machine Translationcited by 6

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.