mlsanityData to 7 Oct 2026

On the Role of Pre-trained Language Models in Word Ordering: A Case Study with BART

arXiv 2204.07367 ↗First posted 15 Apr 2022cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Word ordering is a constrained language generation task taking unordered words as input. Existing work uses linear models and neural networks for the task, yet pre-trained language models have not been studied in word ordering, let alone why they help. We use BART as an instance and show its effectiveness in the task. To explain why BART helps word ordering, we extend analysis with probing and empirically identify that syntactic dependency knowledge in BART is a reliable explanation. We also report performance gains with BART in the related partial tree linearization task, which readily extends our analysis.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 60 references that are in the map. First posted on arXiv 15 Apr 2022.

Builds on

29 of 60 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  11. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
17 more
  1. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  2. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  3. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  4. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  5. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  6. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  7. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  8. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  9. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  10. 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
  11. 2016Modeling Coverage for Neural Machine Translationcited by 110
  12. 2014From Captions to Visual Concepts and Backcited by 109
  13. 2020GPT-too: A language-model-first approach for AMR-to-text generationcited by 38
  14. 2016Word Ordering Without Syntaxcited by 7
  15. 2018Are BLEU and Meaning Representation in Opposition?cited by 5
  16. 2018Neural Transition-based Syntactic Linearizationcited by 3
  17. 2017A Comparison of Neural Models for Word Orderingcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2025Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Abilitycited by 10
  2. 2023Learning to Predict Concept Ordering for Common Sense Generationcited by 3