mlsanityData to 14 Oct 2026

Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings

arXiv 2310.19420 ↗First posted 30 Oct 2023cs.CL cs.LGcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

This paper explores the use of latent bootstrapping, an alternative self-supervision technique, for pretraining language models. Unlike the typical practice of using self-supervision on discrete subwords, latent bootstrapping leverages contextualized embeddings for a richer supervision signal. We conduct experiments to assess how effective this approach is for acquiring linguistic knowledge from limited resources. Specifically, our experiments are based on the BabyLM shared task, which includes pretraining on two small curated corpora and an evaluation on four linguistic benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 39 of its 68 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

39 of 68 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2015Fast R-CNNcited by 2,933
27 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  10. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  11. 2018Neural Network Acceptability Judgmentscited by 958
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  13. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  14. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  15. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  16. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  17. 2019CamemBERT: a Tasty French Language Modelcited by 368
  18. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  19. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  20. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  21. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  22. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  23. 2018A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguisticscited by 64
  24. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  25. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  26. 2020On the importance of pre-training data volume for compact language modelscited by 21
  27. 2023Representation Deficiency in Masked Language Modelingcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Bootstrapping Embeddings for Low Resource Languages
  2. 2025Synthetic bootstrapped pretraining
  3. 2025Latent Principle Discovery for Language Model Self-Improvement
  4. 2024Bootstrap Your Own Context Length
  5. 2023Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures
  6. 2026LLM-Guided Semantic Bootstrapping for Interpretable Text Classification with Tsetlin Machines
  7. 2022An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs
  8. 2024Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping