mlsanityData to 7 Oct 2026

MixUp Training Leads to Reduced Overfitting and Improved Calibration for the Transformer Architecture

arXiv 2102.11402 ↗First posted 22 Feb 2021cs.CL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

MixUp is a computer vision data augmentation technique that uses convex interpolations of input data and their labels to enhance model generalization during training. However, the application of MixUp to the natural language understanding (NLU) domain has been limited, due to the difficulty of interpolating text directly in the input space. In this study, we propose MixUp methods at the Input, Manifold, and sentence embedding levels for the transformer architecture, and apply them to finetune the BERT model for a diverse set of NLU tasks. We find that MixUp can improve model performance, as well as reduce test loss and model calibration error by up to 50%.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 9 of its 14 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

9 of 14 references.

References · 9

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2020Calibration of Pre-trained Transformerscited by 307
  8. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  9. 2019Augmenting Data with Mixup for Sentence Classification: An Empirical Studycited by 125

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 5; 2024: 2; 2025: 3.

11 papers.

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  3. 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
  4. 2024Revisiting Confidence Estimation: Towards Reliable Failure Predictioncited by 9
  5. 2023Bag of Tricks for In-Distribution Calibration of Pretrained Transformerscited by 7
  6. 2023TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimationcited by 4
  7. 2022Enhancing Cross-lingual Prompting with Dual Prompt Augmentationcited by 3
  8. 2024SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipecited by 3
  9. 2025LM-mixup: Text Data Augmentation via Language Model based Mixupcited by 3
  10. 2025h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objectivecited by 2
  11. 2025Mitigating Shortcut Learning with InterpoLated Learningcited by 2