mlsanityData to 14 Oct 2026

GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation

arXiv 2211.10330 ↗First posted 18 Nov 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

We introduce GENIUS: a conditional text generation model using sketches as input, which can fill in the missing contexts for a given sketch (key information consisting of textual spans, phrases, or words, concatenated by mask tokens). GENIUS is pre-trained on a large-scale textual corpus with a novel reconstruction from sketch objective using an extreme and selective masking strategy, enabling it to generate diverse and high-quality texts given sketches. Comparison with other competitive conditional language models (CLMs) reveals the superiority of GENIUS's text generation quality. We further show that GENIUS can be used as a strong and ready-to-use data augmentation tool for various natural language processing (NLP) tasks. Most existing textual data augmentation methods are either too conservative, by making small changes to the original text, or too aggressive, by creating entirely new samples. With GENIUS, we propose GeniusAug, which first extracts the target-aware sketches from the original training set and then generates new samples based on the sketches. Empirical experiments on 6 text classification datasets show that GeniusAug significantly improves the models' performance in both in-distribution (ID) and out-of-distribution (OOD) settings. We also demonstrate the effectiveness of GeniusAug on named entity recognition (NER) and machine reading comprehension (MRC) tasks. (Code and models are publicly available at https://github.com/microsoft/SCGLab and https://github.com/beyondguo/genius)

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 47 of its 68 references that are in the map. First posted on arXiv 18 Nov 2022.

Builds on

47 of 68 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
35 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  3. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  4. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  5. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  6. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  7. 2021A Survey of Data Augmentation Approaches for NLPcited by 409
  8. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  9. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  10. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  11. 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 234
  12. 2018A Note on the Inception Scorecited by 226
  13. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 187
  14. 2019Not Enough Data? Deep Learning to the Rescue!cited by 175
  15. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  16. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  17. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  18. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 128
  19. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125
  20. 2020Enabling Language Models to Fill in the Blankscited by 122
  21. 2018Conditional BERT Contextual Augmentationcited by 107
  22. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  23. 2020An Analysis of Simple Data Augmentation for Named Entity Recognitioncited by 76
  24. 2020Blank Language Modelscited by 43
  25. 2020Paraphrase Augmented Task-Oriented Dialog Generationcited by 43
  26. 2021MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NERcited by 42
  27. 2020GenAug: Data Augmentation for Finetuning Text Generatorscited by 41
  28. 2020Empirical Analysis of Unlabeled Entity Problem in Named Entity Recognitioncited by 35
  29. 2019Text Infillingcited by 32
  30. 2021FlipDA: Effective and Robust Data Augmentation for Few-Shot Learningcited by 32
  31. 2021Parallel Refinements for Lexically Constrained Text Generation with BARTcited by 24
  32. 2020Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Spacecited by 19
  33. 2019TIGS: An Inference Algorithm for Text Infilling with Gradient Searchcited by 10
  34. 2019XL-Editor: Post-editing Sentences with XLNetcited by 5
  35. 2022Selective Text Augmentation with Word Roles for Low-Resource Text Classificationcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 8.

11 papers.

  1. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  2. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  3. 2024CoDa: Constrained Generation based Data Augmentation for Low-Resource NLPcited by 9
  4. 2024ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptionscited by 8
  5. 2023CMD: a framework for Context-aware Model self-Detoxificationcited by 6
  6. 2024An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classificationcited by 5
  7. 2024Annotating FrameNet via Structure-Conditioned Language Generationcited by 2
  8. 2024Data Augmentation of Multi-turn Psychological Dialogue via Knowledge-driven Progressive Thought Promptingcited by 2
  9. 2022A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Taskscited by 1
  10. 2024Sample Design Engineering: An Empirical Study of What Makes Good Downstream Fine-Tuning Samples for LLMscited by 1
  11. 2024Topic-to-essay generation with knowledge-based content selectioncited by 0