mlsanityData to 7 Oct 2026

SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation

arXiv 2310.00796 ↗First posted 1 Oct 2023cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Strong inductive biases enable learning from little data and help generalization outside of the training distribution. Popular neural architectures such as Transformers lack strong structural inductive biases for seq2seq NLP tasks on their own. Consequently, they struggle with systematic generalization beyond the training distribution, e.g. with extrapolating to longer inputs, even when pre-trained on large amounts of text. We show how a structural inductive bias can be efficiently injected into a seq2seq model by pre-training it to simulate structural transformations on synthetic data. Specifically, we inject an inductive bias towards Finite State Transducers (FSTs) into a Transformer by pre-training it to simulate FSTs given their descriptions. Our experiments show that our method imparts the desired inductive bias, resulting in improved systematic generalization and better few-shot learning for FST-like tasks. Our analysis shows that fine-tuned models accurately capture the state dynamics of the unseen underlying FSTs, suggesting that the simulation process is internalized by the fine-tuned model.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 1 Oct 2023.

Builds on

35 of 52 references.

References · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  6. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  7. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  8. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  9. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  10. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  11. 2022Neural Networks and the Chomsky Hierarchycited by 214
  12. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
23 more
  1. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  2. 2014Learning to Executecited by 105
  3. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  4. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  5. 2022ByT5 model for massively multilingual grapheme-to-phoneme conversioncited by 49
  6. 2021LIME: Learning Inductive Bias for Primitives of Mathematical Reasoningcited by 46
  7. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 40
  8. 2021On Compositional Generalization of Neural Machine Translationcited by 34
  9. 2023Injecting structural hints: Using language models to study inductive biases in language learningcited by 33
  10. 2020Compositional Generalization via Semantic Taggingcited by 32
  11. 2021Sequence-to-Sequence Learning with Latent Neural Grammarscited by 30
  12. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
  13. 2020Universal linguistic inductive biases via meta-learningcited by 28
  14. 2020Hierarchical Poset Decoding for Compositional Generalization in Languagecited by 26
  15. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 25
  16. 2019Exact Hard Monotonic Attention for Character-Level Transductioncited by 22
  17. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  18. 2022Static Prediction of Runtime Errors by Learning to Execute Programs with External Resource Descriptionscited by 11
  19. 2022What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environmentcited by 9
  20. 2018Parser Training with Heterogeneous Treebankscited by 8
  21. 2016Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learningcited by 5
  22. 2023Compositional Generalization without Trees using Multiset Tagging and Latent Permutationscited by 5
  23. 2017SyGuS-Comp 2017: Results and Analysiscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  2. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  3. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  4. 2024Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformationscited by 4
  5. 2025Transfer of Structural Knowledge from Synthetic Languagescited by 3
  6. 2026Procedural Core: A Compact Recurrent Initialization for Vision Transformerscited by 0
  7. 2026Lasting Effects of Abstract Pretraining Beyond Perplexitycited by 0