mlsanityData to 7 Oct 2026

How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task

arXiv 2111.09612 ↗First posted 18 Nov 2021cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Despite their success, modern language models are fragile. Even small changes in their training pipeline can lead to unexpected results. We study this phenomenon by examining the robustness of ALBERT (arXiv:1909.11942) in combination with Stochastic Weight Averaging (SWA) (arXiv:1803.05407) -- a cheap way of ensembling -- on a sentiment analysis task (SST-2). In particular, we analyze SWA's stability via CheckList criteria (arXiv:2005.04118), examining the agreement on errors made by models differing only in their random seed. We hypothesize that SWA is more stable because it ensembles model snapshots taken along the gradient descent trajectory. We quantify stability by comparing the models' mistakes with Fleiss' Kappa (Fleiss, 1971) and overlap ratio scores. We find that SWA reduces error rates in general; yet the models still suffer from their own distinct biases (according to CheckList).

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

19 of 30 references.

References · 19

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  5. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  6. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  7. 2021Robustness Gym: Unifying the NLP Evaluation Landscapecited by 107
  8. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  9. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  10. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 69
  11. 2021The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Modelscited by 63
  12. 2021Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Levelcited by 34
7 more
  1. 2020Robustness Testing of Language Understanding in Task-Oriented Dialogcited by 32
  2. 2019On Model Stability as a Function of Random Seedcited by 31
  3. 2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
  4. 2020Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillationcited by 25
  5. 2021BiasFinder: Metamorphic Test Generation to Uncover Bias for Sentiment Analysis Systemscited by 20
  6. 2021Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanationscited by 15
  7. 2021On the Impact of Random Seeds on the Fairness of Clinical Classifierscited by 8

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2.

5 papers.

  1. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  2. 2023Measuring the Instability of Fine-Tuningcited by 7
  3. 2023Uncertainty-Aware Natural Language Inference with Stochastic Weight Averagingcited by 4
  4. 2024Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolationcited by 2
  5. 2024You Shall Know a Tool by the Traces it Leaves: The Predictability of Sentiment Analysis Toolscited by 0