# Optimizing Deeper Transformers on Small Datasets

arXiv: https://arxiv.org/abs/2012.15355 · HTML: https://mlsanity.com/paper/2012.15355/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2020.

## Built on by (most cited first, 30 of 42)

- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL](https://mlsanity.com/paper/2302.05965/) (2023)
- [DeepNet: Scaling Transformers to 1,000 Layers](https://mlsanity.com/paper/2203.00555/) (2022)
- [LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations](https://mlsanity.com/paper/2106.01093/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers](https://mlsanity.com/paper/2303.01610/) (2023)
- [A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions](https://mlsanity.com/paper/2208.13629/) (2022)
- [Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect](https://mlsanity.com/paper/2208.10099/) (2022)
- [Transformers in Time-series Analysis: A Tutorial](https://mlsanity.com/paper/2205.01138/) (2022)
- [SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL](https://mlsanity.com/paper/2111.00653/) (2021)
- [Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful](https://mlsanity.com/paper/2507.07101/) (2025)
- [Importance of Synthesizing High-quality Data for Text-to-SQL Parsing](https://mlsanity.com/paper/2212.08785/) (2022)
- [HIE-SQL: History Information Enhanced Network for Context-Dependent Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2203.07376/) (2022)
- [Semantic Enhanced Text-to-SQL Parsing via Iteratively Learning Schema Linking Graph](https://mlsanity.com/paper/2208.03903/) (2022)
- [mRAT-SQL+GAP:A Portuguese Text-to-SQL Transformer](https://mlsanity.com/paper/2110.03546/) (2021)
- [Hierarchical Neural Data Synthesis for Semantic Parsing](https://mlsanity.com/paper/2112.02212/) (2021)
- [From Words to Code: Harnessing Data for Program Synthesis from Natural Language](https://mlsanity.com/paper/2305.01598/) (2023)
- [Dynamic Linear Transformer for 3D Biomedical Image Segmentation](https://mlsanity.com/paper/2206.00771/) (2022)
- [Relation Aware Semi-autoregressive Semantic Parsing for NL2SQL](https://mlsanity.com/paper/2108.00804/) (2021)
- [Diverse Parallel Data Synthesis for Cross-Database Adaptation of Text-to-SQL Parsers](https://mlsanity.com/paper/2210.16613/) (2022)
- [Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation](https://mlsanity.com/paper/2309.03019/) (2023)
- [Relaxed Attention for Transformer Models](https://mlsanity.com/paper/2209.09735/) (2022)
- [Mass Spectra Prediction with Structural Motif-based Graph Neural Networks](https://mlsanity.com/paper/2306.16085/) (2023)
- [Delving into Differentially Private Transformer](https://mlsanity.com/paper/2405.18194/) (2024)
- [Inferring the past: a combined CNN-LSTM deep learning framework to fuse satellites for historical inundation mapping](https://mlsanity.com/paper/2305.00640/) (2023)
- [Generalization Gap in Data Augmentation: Insights from Illumination](https://mlsanity.com/paper/2404.07514/) (2024)
- [Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide](https://mlsanity.com/paper/2411.09539/) (2024)
- [Structured Case-based Reasoning for Inference-time Adaptation of Text-to-SQL parsers](https://mlsanity.com/paper/2301.04110/) (2023)

## Abstract (arXiv)

It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train 48 layers of transformers, comprising 24 fine-tuned layers from pre-trained RoBERTa and 24 relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider. We achieve this by deriving a novel Data-dependent Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the prior T-Fixup work. Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.