# Revisiting the Markov Property for Machine Translation

arXiv: https://arxiv.org/abs/2402.02084 · HTML: https://mlsanity.com/paper/2402.02084/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Method](https://mlsanity.com/paper/2310.00305/) (2023)
- [Fast Structured Decoding for Sequence Models](https://mlsanity.com/paper/1910.11555/) (2019)
- [Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2106.05093/) (2021)
- [Efficient Second-Order TreeCRF for Neural Dependency Parsing](https://mlsanity.com/paper/2005.00975/) (2020)
- [Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF](https://mlsanity.com/paper/1704.01314/) (2017)
- [Cascaded Text Generation with Markov Transformers](https://mlsanity.com/paper/2006.01112/) (2020)
- [Faster Transformer Decoding: N-gram Masked Self-Attention](https://mlsanity.com/paper/2001.04589/) (2020)

## Built on by (most cited first, 2 of 2)

- [Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2406.09136/) (2024)
- [Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs](https://mlsanity.com/paper/2505.19155/) (2025)

## Abstract (arXiv)

In this paper, we re-examine the Markov property in the context of neural machine translation. We design a Markov Autoregressive Transformer~(MAT) and undertake a comprehensive assessment of its performance across four WMT benchmarks. Our findings indicate that MAT with an order larger than 4 can generate translations with quality on par with that of conventional autoregressive transformers. In addition, counter-intuitively, we also find that the advantages of utilizing a higher-order MAT do not specifically contribute to the translation of longer sentences.