mlsanityData to 7 Oct 2026

What do Neural Machine Translation Models Learn about Morphology?

arXiv 1704.03471 ↗First posted 11 Apr 2017cs.CLcited by 165

Not on the map. Why?

Abstract (arXiv)

Neural machine translation (MT) models obtain state-of-the-art performance while maintaining a simple, end-to-end architecture. However, little is known about what these models learn about source and target languages during the training process. In this work, we analyze the representations learned by neural MT models at various levels of granularity and empirically evaluate the quality of the representations for learning morphology through extrinsic part-of-speech and morphological tagging tasks. We conduct a thorough investigation along several parameters: word-based vs. character-based representations, depth of the encoding layer, the identity of the target language, and encoder vs. decoder representations. Our data-driven, quantitative evaluation sheds light on important aspects in the neural MT system and its ability to capture word structure.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 36; 2023: 27; 2024: 19; 2025: 22; 2026: 19.

165 papers.

Sort:

By area:

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  4. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  5. 2021Fast Model Editing at Scalecited by 491
  6. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  7. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  8. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  9. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
  10. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  11. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  12. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
38 more
  1. 2022Discovering Latent Concepts Learned in BERTcited by 56
  2. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  3. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  4. 2021Probing BERT in Hyperbolic Spacescited by 50
  5. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  6. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  7. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44
  8. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  9. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 39
  10. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  11. 2023The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Modelscited by 37
  12. 2021What do pre-trained code models know about code?cited by 36
  13. 2022The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlativecited by 35
  14. 2024Locating and Editing Factual Associations in Mambacited by 35
  15. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  16. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  17. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  18. 2022Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsingcited by 30
  19. 2021Grounding Representation Similarity with Statistical Testingcited by 28
  20. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27
  21. 2023Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignmentcited by 27
  22. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  23. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 26
  24. 2024Mechanistic?cited by 25
  25. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  26. 2021How much pretraining data do language models need to learn syntax?cited by 21
  27. 2022Discovering Salient Neurons in Deep NLP Modelscited by 21
  28. 2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
  29. 2021Memorization and Generalization in Neural Code Intelligence Modelscited by 17
  30. 2022AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language modelscited by 16
  31. 2023Can LLMs facilitate interpretation of pre-trained language models?cited by 15
  32. 2024Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Modelscited by 15
  33. 2021HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognitioncited by 14
  34. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 14
  35. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  36. 2025The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Modelscited by 14
  37. 2025Large Language Models Encode Semantics and Alignment in Linearly Separable Representationscited by 14
  38. 2021How Reliable are Model Diagnostics?cited by 13

50 most cited of 165 All 165 (JSON)