# Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning

arXiv: https://arxiv.org/abs/2102.03554 · HTML: https://mlsanity.com/paper/2102.03554/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 6 Feb 2021.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Curriculum Learning and Minibatch Bucketing in Neural Machine Translation](https://mlsanity.com/paper/1707.09533/) (2017)
- [Norm-Based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2006.02014/) (2020)
- [Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/1905.05816/) (2019)
- [An Empirical Exploration of Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1811.00739/) (2018)
- [Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence](https://mlsanity.com/paper/2005.01096/) (2020)
- [DART: A Lightweight Quality-Suggestive Data-to-Text Annotation Tool](https://mlsanity.com/paper/2010.04141/) (2020)
- [Unsupervised Pidgin Text Generation By Pivoting English Data and Self-Training](https://mlsanity.com/paper/2003.08272/) (2020)
- [Neobility at SemEval-2017 Task 1: An Attention-based Sentence Similarity Model](https://mlsanity.com/paper/1703.05465/) (2017)
- [Safe Handover in Mixed-Initiative Control for Cyber-Physical Systems](https://mlsanity.com/paper/2010.10967/) (2020)

## Built on by (most cited first, 30 of 34)

- [Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples](https://mlsanity.com/paper/2502.09650/) (2025)
- [LeanAgent: Lifelong Learning for Formal Theorem Proving](https://mlsanity.com/paper/2410.06209/) (2024)
- [RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training](https://mlsanity.com/paper/2509.21009/) (2025)
- [Deep Spatially and Temporally Aware Similarity Computation for Road Network Constrained Trajectories](https://mlsanity.com/paper/2112.09339/) (2021)
- [Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learning](https://mlsanity.com/paper/2405.07490/) (2024)
- [Denoising Task Difficulty-based Curriculum for Training Diffusion Models](https://mlsanity.com/paper/2403.10348/) (2024)
- [On Training Instance Selection for Few-Shot Neural Text Generation](https://mlsanity.com/paper/2107.03176/) (2021)
- [An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Models](https://mlsanity.com/paper/2203.09486/) (2022)
- [Time-Aware Ancient Chinese Text Translation and Inference](https://mlsanity.com/paper/2107.03179/) (2021)
- [Data Efficacy for Language Model Training](https://mlsanity.com/paper/2506.21545/) (2025)
- [Too Much Information: Keeping Training Simple for BabyLMs](https://mlsanity.com/paper/2311.01955/) (2023)
- [HiCuLR: Hierarchical Curriculum Learning for Rhetorical Role Labeling of Legal Documents](https://mlsanity.com/paper/2409.18647/) (2024)
- [The SelectGen Challenge: Finding the Best Training Samples for Few-Shot Neural Text Generation](https://mlsanity.com/paper/2108.06614/) (2021)
- [Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction](https://mlsanity.com/paper/2204.10360/) (2022)
- [LLM Attributor: Interactive Visual Attribution for LLM Generation](https://mlsanity.com/paper/2404.01361/) (2024)
- [Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis](https://mlsanity.com/paper/2405.14277/) (2024)
- [In-sample Curriculum Learning by Sequence Completion for Natural Language Generation](https://mlsanity.com/paper/2211.11297/) (2022)
- [Curriculum-guided Abstractive Summarization for Mental Health Online Posts](https://mlsanity.com/paper/2302.00954/) (2023)
- [GIT-CXR: End-to-End Transformer for Chest X-Ray Report Generation](https://mlsanity.com/paper/2501.02598/) (2025)
- [Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text](https://mlsanity.com/paper/2502.12953/) (2025)
- [Demystifying Data Organization for Enhanced LLM Training](https://mlsanity.com/paper/2605.30334/) (2026)
- [Adaptive Multi-Corpora Language Model Training for Speech Recognition](https://mlsanity.com/paper/2211.05121/) (2022)
- [A Severity-Based Curriculum Learning Strategy for Arabic Medical Text Generation](https://mlsanity.com/paper/2604.06365/) (2026)
- [Medical Scientific Table-to-Text Generation with Human-in-the-Loop under the Data Sparsity Constraint](https://mlsanity.com/paper/2205.12368/) (2022)
- [High-Resource Methodological Bias in Low-Resource Investigations](https://mlsanity.com/paper/2211.07534/) (2022)
- [Data Debugging is NP-hard for Classifiers Trained with SGD](https://mlsanity.com/paper/2408.01365/) (2024)
- [Curriculum Learning for Cross-Lingual Data-to-Text Generation With Noisy Data](https://mlsanity.com/paper/2412.13484/) (2024)
- [Learning from Impairment: Leveraging Insights from Clinical Linguistics in Language Modelling Research](https://mlsanity.com/paper/2412.15785/) (2024)
- [CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimization](https://mlsanity.com/paper/2502.18532/) (2025)
- [CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learning](https://mlsanity.com/paper/2601.05858/) (2026)

## Abstract (arXiv)

Recent advancements in data-to-text generation largely take on the form of neural end-to-end systems. Efforts have been dedicated to improving text generation systems by changing the order of training samples in a process known as curriculum learning. Past research on sequence-to-sequence learning showed that curriculum learning helps to improve both the performance and convergence speed. In this work, we delve into the same idea surrounding the training samples consisting of structured data and text pairs, where at each update, the curriculum framework selects training samples based on the model's competence. Specifically, we experiment with various difficulty metrics and put forward a soft edit distance metric for ranking training samples. Our benchmarks show faster convergence speed where training time is reduced by 38.7% and performance is boosted by 4.84 BLEU.