mlsanityData to 13 Oct 2026

Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks

arXiv 1801.07772 ↗First posted 23 Jan 2018cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

While neural machine translation (NMT) models provide improved translation quality in an elegant, end-to-end framework, it is less clear what they learn about language. Recent work has started evaluating the quality of vector representations learned by NMT models on morphological and syntactic tasks. In this paper, we investigate the representations learned at different layers of NMT encoders. We train NMT systems on parallel data and use the trained models to extract features for training a classifier on two tasks: part-of-speech and semantic tagging. We then measure the performance of the classifier as a proxy to the quality of the original NMT model for the given task. Our quantitative analysis yields interesting insights regarding representation learning in NMT models. For instance, we find that higher layers are better at learning semantics while lower layers tend to be better for part-of-speech tagging. We also observe little effect of the target language on source-side representations, especially with higher quality NMT models.

Key facts. Cited by 56 papers in the map (data through 13 Oct 2026). First posted on arXiv 23 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 18; 2022: 12; 2023: 8; 2024: 5; 2025: 10; 2026: 3.

56 papers.

Sort:

By area:

  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 850
  2. 2023Explainability for Large Language Models: A Surveycited by 333
  3. 2023The geometry of hidden representations of large transformer modelscited by 109
  4. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  5. 2021Meta-Learning to Compositionally Generalizecited by 62
  6. 2022Discovering Latent Concepts Learned in BERTcited by 56
  7. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  8. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  9. 2022Prompting Language Models for Linguistic Structurecited by 39
  10. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  11. 2021On Compositional Generalization of Neural Machine Translationcited by 34
  12. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
38 more
  1. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  2. 2024Mechanistic?cited by 25
  3. 2021A Bayesian Framework for Information-Theoretic Probingcited by 22
  4. 2022Discovering Salient Neurons in Deep NLP Modelscited by 21
  5. 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoderscited by 21
  6. 2022Naturalistic Causal Probing for Morpho-Syntaxcited by 19
  7. 2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
  8. 2021How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impactcited by 18
  9. 2023Can LLMs facilitate interpretation of pre-trained language models?cited by 15
  10. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 15
  11. 2024Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Modelscited by 15
  12. 2023Operationalising Representation in Natural Language Processingcited by 14
  13. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  14. 2021Micromodels for Efficient, Explainable, and Reusable Systems: A Case Study on Mental Healthcited by 13
  15. 2021Does injecting linguistic structure into language models lead to better alignment with brain recordings?cited by 11
  16. 2021Bird's Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approachcited by 9
  17. 2021A multilabel approach to morphosyntactic probingcited by 8
  18. 2023Morphosyntactic probing of multilingual BERT modelscited by 8
  19. 2021Effect of Post-processing on Contextualized Word Representationscited by 7
  20. 2022Probing via Promptingcited by 7
  21. 2025Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translationcited by 7
  22. 2025Different types of syntactic agreement recruit the same units within large language modelscited by 7
  23. 2021Morph Call: Probing Morphosyntactic Content of Multilingual Transformerscited by 5
  24. 2025Self-Regularization with Sparse Autoencoders for Controllable LLM-based Classificationcited by 5
  25. 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcyclingcited by 5
  26. 2021Fine-grained Interpretation and Causation Analysis in Deep NLP Modelscited by 4
  27. 2022Understanding Domain Learning in Language Models Through Subpopulation Analysiscited by 4
  28. 2022Is neural language acquisition similar to natural? A chronological probing studycited by 3
  29. 2022Probing with Noise: Unpicking the Warp and Weft of Embeddingscited by 3
  30. 2025TRACE: Training and Inference-Time Interpretability Analysis for Language Modelscited by 3
  31. 2025Towards Transparent AI: A Survey on Explainable Language Modelscited by 3
  32. 2021Hierarchical Transformer for Multilingual Machine Translationcited by 2
  33. 2021Variation and generality in encoding of syntactic anomaly information in sentence embeddingscited by 2
  34. 2022Log-linear Guardedness and its Implicationscited by 1
  35. 2023NxPlain: Web-based Tool for Discovery of Latent Conceptscited by 1
  36. 2024Analyzing Narrative Processing in Large Language Models (LLMs): Using GPT4 to test BERTcited by 1
  37. 2025A Representation Level Analysis of NMT Model Robustness to Grammatical Errorscited by 1
  38. 2026Prompting Underestimates LLM Capability for Time Series Classificationcited by 1

50 most cited of 56 All 56 (JSON)