mlsanityData to 7 Oct 2026

Gradient Ascent Post-training Enhances Language Model Generalization

arXiv 2306.07052 ↗First posted 12 Jun 2023cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

In this work, we empirically show that updating pretrained LMs (350M, 1.3B, 2.7B) with just a few steps of Gradient Ascent Post-training (GAP) on random, unlabeled text corpora enhances its zero-shot generalization capabilities across diverse NLP tasks. Specifically, we show that GAP can allow LMs to become comparable to 2-3x times larger LMs across 12 different NLP tasks. We also show that applying GAP on out-of-distribution corpora leads to the most reliable performance improvements. Our findings indicate that GAP can be a promising method for improving the generalization capability of LMs without any task-specific fine-tuning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

30 of 41 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
18 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  8. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  9. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  10. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  11. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  12. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  13. 2022A Contrastive Framework for Neural Text Generationcited by 224
  14. 2021Internet-Augmented Dialogue Generationcited by 217
  15. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
  16. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  17. 2022When Do Flat Minima Optimizers Work?cited by 78
  18. 2022COPA-SSE: Semi-structured Explanations for Commonsense Reasoningcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  2. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methodscited by 38
  3. 2024RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Modelscited by 21