mlsanityData to 14 Oct 2026

Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures

arXiv 2305.18915 ↗First posted 30 May 2023cs.CL cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

In this work we build upon negative results from an attempt at language modeling with predicted semantic structure, in order to establish empirical lower bounds on what could have made the attempt successful. More specifically, we design a concise binary vector representation of semantic structure at the lexical level and evaluate in-depth how good an incremental tagger needs to be in order to achieve better-than-baseline performance with an end-to-end semantic-bootstrapping language model. We envision such a system as consisting of a (pretrained) sequential-neural component and a hierarchical-symbolic component working together to generate text with low surprisal and high linguistic interpretability. We find that (a) dimensionality of the semantic vector representation can be dramatically reduced without losing its main advantages and (b) lower bounds on prediction quality cannot be established via a single score alone, but need to take the distributions of signal and noise into account.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 37 of its 70 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

37 of 70 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  6. 2019Designing and Interpreting Probes with Control Taskscited by 640
  7. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  8. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  9. 2021Causal Abstractions of Neural Networkscited by 422
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  11. 2018Pyro: Deep Universal Probabilistic Programmingcited by 380
  12. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
25 more
  1. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  2. 2018Model compression via distillation and quantizationcited by 289
  3. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  4. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 119
  5. 2016Recurrent Neural Network Grammarscited by 109
  6. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  7. 2022Interpreting Language Models with Contrastive Explanationscited by 92
  8. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  9. 2016Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representationscited by 55
  10. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  11. 2019Augmenting Neural Networks with First-order Logiccited by 51
  12. 2019Probing What Different NLP Tasks Teach Machines about Function Word Comprehensioncited by 44
  13. 2018Finding Syntax in Human Encephalography with Beam Searchcited by 41
  14. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 41
  15. 2020Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigationcited by 34
  16. 2022SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Featurescited by 33
  17. 2017Effective Inference for Generative Neural Parsingcited by 25
  18. 2021Structural Guidance for Transformer Language Modelscited by 25
  19. 2020Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modelingcited by 22
  20. 2020Posterior Control of Blackbox Generationcited by 16
  21. 2020(Re)construing Meaning in NLPcited by 16
  22. 2020Supertagging the Long Tail with Tree-Structured Decoding of Complex Categoriescited by 11
  23. 2021Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modelingcited by 10
  24. 2020What are the Goals of Distributional Semantics?cited by 8
  25. 2023Understanding Natural Language Understanding Systems. A Critical Analysiscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Deep networks learn to parse uniform-depth context-free languages from local statistics
  2. 2020Aligning the Pretraining and Finetuning Objectives of Language Models
  3. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility
  4. 2023Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings
  5. 2024Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations
  6. 2025A Markov Categorical Framework for Language Modeling
  7. 2024From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars
  8. 2026LLM-Guided Semantic Bootstrapping for Interpretable Text Classification with Tsetlin Machines