# On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference

arXiv: https://arxiv.org/abs/1804.09779 · HTML: https://mlsanity.com/paper/1804.09779/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2018.

## Built on by (most cited first, 5 of 5)

- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference](https://mlsanity.com/paper/2306.09918/) (2023)
- [Variation and generality in encoding of syntactic anomaly information in sentence embeddings](https://mlsanity.com/paper/2111.06644/) (2021)
- [Survey of NLU Benchmarks Diagnosing Linguistic Phenomena: Why not Standardize Diagnostics Benchmarks?](https://mlsanity.com/paper/2507.20419/) (2025)
- [E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding](https://mlsanity.com/paper/2609.33530/) (2026)

## Abstract (arXiv)

We propose a process for investigating the extent to which sentence representations arising from neural machine translation (NMT) systems encode distinct semantic phenomena. We use these representations as features to train a natural language inference (NLI) classifier based on datasets recast from existing semantic annotations. In applying this process to a representative NMT system, we find its encoder appears most suited to supporting inferences at the syntax-semantics interface, as compared to anaphora resolution requiring world-knowledge. We conclude with a discussion on the merits and potential deficiencies of the existing process, and how it may be improved and extended as a broader framework for evaluating semantic coverage.