mlsanityData to 14 Oct 2026

Spiral Language Modeling

arXiv 2112.10543 ↗First posted 20 Dec 2021cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In almost all text generation applications, word sequences are constructed in a left-to-right (L2R) or right-to-left (R2L) manner, as natural language sentences are written either L2R or R2L. However, we find that the natural language written order is not essential for text generation. In this paper, we propose Spiral Language Modeling (SLM), a general approach that enables one to construct natural language sentences beyond the L2R and R2L order. SLM allows one to form natural language text by starting from an arbitrary token inside the result text and expanding the rest tokens around the selected ones. It makes the decoding order a new optimization objective besides the language model perplexity, which further improves the diversity and quality of the generated text. Furthermore, SLM makes it possible to manipulate the text construction process by selecting a proper starting token. SLM also introduces generation orderings as additional regularization to improve model robustness in low-resource scenarios. Experiments on 8 widely studied Neural Machine Translation (NMT) tasks show that SLM is constantly effective with up to 4.7 BLEU increase comparing to the conventional L2R decoding approach.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 20 Dec 2021.

Builds on

22 of 27 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  6. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 307
  7. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  8. 2016Neural Autoregressive Distribution Estimationcited by 95
  9. 2016Edinburgh Neural Machine Translation Systems for WMT 16cited by 76
  10. 2013A Deep and Tractable Density Estimatorcited by 68
  11. 2019Encoding word order in complex embeddingscited by 53
  12. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
10 more
  1. 2016Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversationcited by 35
  2. 2021AutoDropout: Learning Dropout Patterns to Regularize Deep Networkscited by 35
  3. 2019Synchronous Bidirectional Neural Machine Translationcited by 27
  4. 2018Asynchronous Bidirectional Decoding for Neural Machine Translationcited by 21
  5. 2017The University of Edinburgh's Neural MT Systems for WMT17cited by 16
  6. 2017Towards Decoding as Continuous Optimization in Neural Machine Translationcited by 15
  7. 2018A Syntactically Constrained Bidirectional-Asynchronous Approach for Emotional Conversation Generationcited by 11
  8. 2015Backward and Forward Language Modeling for Constrained Sentence Generationcited by 9
  9. 2018BFGAN: Backward and Forward Generative Adversarial Networks for Lexically Constrained Sentence Generationcited by 9
  10. 2019Sequence Generation: From Both Sides to the Middlecited by 5

Built on by

Not cited yet.