# Optimal word order for non-causal text generation with Large Language Models: the Spanish case

arXiv: https://arxiv.org/abs/2502.14451 · HTML: https://mlsanity.com/paper/2502.14451/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 57 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators](https://mlsanity.com/paper/2108.00946/) (2021)
- [Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction](https://mlsanity.com/paper/2005.00987/) (2020)
- [Blank Language Models](https://mlsanity.com/paper/2002.03079/) (2020)
- [MarIA: Spanish Language Models](https://mlsanity.com/paper/2107.07253/) (2021)
- [Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment](https://mlsanity.com/paper/2005.00165/) (2020)
- [Comparing the Pearson and Spearman Correlation Coefficients Across Distributions and Sample Sizes: A Tutorial Using Simulations and Empirical Data](https://mlsanity.com/paper/2408.15979/) (2024)
- [A Financial Service Chatbot based on Deep Bidirectional Transformers](https://mlsanity.com/paper/2003.04987/) (2020)
- [You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models](https://mlsanity.com/paper/2210.07135/) (2022)

## Built on by (most cited first, 2 of 2)

- [Mixture of Routers](https://mlsanity.com/paper/2503.23362/) (2025)
- [Decoding Order Matters in Autoregressive Speech Synthesis](https://mlsanity.com/paper/2601.08450/) (2026)

## Abstract (arXiv)

Natural Language Generation (NLG) popularity has increased owing to the progress in Large Language Models (LLMs), with zero-shot inference capabilities. However, most neural systems utilize decoder-only causal (unidirectional) transformer models, which are effective for English but may reduce the richness of languages with less strict word order, subject omission, or different relative clause attachment preferences. This is the first work that analytically addresses optimal text generation order for non-causal language models. We present a novel Viterbi algorithm-based methodology for maximum likelihood word order estimation. We analyze the non-causal most-likelihood order probability for NLG in Spanish and, then, the probability of generating the same phrases with Spanish causal NLG. This comparative analysis reveals that causal NLG prefers English-like SVO structures. We also analyze the relationship between optimal generation order and causal left-to-right generation order using Spearman's rank correlation. Our results demonstrate that the ideal order predicted by the maximum likelihood estimator is not closely related to the causal order and may be influenced by the syntactic structure of the target sentence.