# What do Neural Machine Translation Models Learn about Morphology?

arXiv: https://arxiv.org/abs/1704.03471 · HTML: https://mlsanity.com/paper/1704.03471/

## Key facts

- Cited by 166 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2017.

## Built on by (most cited first, 30 of 166)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [SAEs Are Good for Steering -- If You Select the Right Features](https://mlsanity.com/paper/2505.20063/) (2025)
- [Discovering Latent Concepts Learned in BERT](https://mlsanity.com/paper/2205.07237/) (2022)
- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure](https://mlsanity.com/paper/2101.00387/) (2021)
- [Probing BERT in Hyperbolic Spaces](https://mlsanity.com/paper/2104.03869/) (2021)
- [Do Transformers Parse while Predicting the Masked Word?](https://mlsanity.com/paper/2303.08117/) (2023)
- [Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era](https://mlsanity.com/paper/2403.08946/) (2024)
- [Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder](https://mlsanity.com/paper/2304.04052/) (2023)
- [Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks](https://mlsanity.com/paper/2201.09745/) (2022)
- [Estimating Knowledge in Large Language Models Without Generating a Single Token](https://mlsanity.com/paper/2406.12673/) (2024)
- [The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models](https://mlsanity.com/paper/2311.05928/) (2023)
- [Analyzing Encoded Concepts in Transformer Language Models](https://mlsanity.com/paper/2206.13289/) (2022)
- [Locating and Editing Factual Associations in Mamba](https://mlsanity.com/paper/2404.03646/) (2024)
- [What do pre-trained code models know about code?](https://mlsanity.com/paper/2108.11308/) (2021)
- [The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative](https://mlsanity.com/paper/2210.13181/) (2022)
- [Distilling Linguistic Context for Language Model Compression](https://mlsanity.com/paper/2109.08359/) (2021)
- [How transfer learning impacts linguistic knowledge in deep NLP models?](https://mlsanity.com/paper/2105.15179/) (2021)
- [Local Interpretations for Explainable Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.11072/) (2021)
- [Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing](https://mlsanity.com/paper/2206.14017/) (2022)

## Abstract (arXiv)

Neural machine translation (MT) models obtain state-of-the-art performance while maintaining a simple, end-to-end architecture. However, little is known about what these models learn about source and target languages during the training process. In this work, we analyze the representations learned by neural MT models at various levels of granularity and empirically evaluate the quality of the representations for learning morphology through extrinsic part-of-speech and morphological tagging tasks. We conduct a thorough investigation along several parameters: word-based vs. character-based representations, depth of the encoding layer, the identity of the target language, and encoder vs. decoder representations. Our data-driven, quantitative evaluation sheds light on important aspects in the neural MT system and its ability to capture word structure.