mlsanityData to 14 Oct 2026

Consistency Regularization for Cross-Lingual Fine-Tuning

arXiv 2106.08226 ↗First posted 15 Jun 2021cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others. In this work, we propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation. In addition, we employ model consistency to regularize the models trained with two augmented versions of the same training set. Experimental results on the XTREME benchmark show that our method significantly improves cross-lingual fine-tuning across various tasks, including text classification, question answering, and sequence labeling.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

35 of 43 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  5. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  6. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  11. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  12. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
23 more
  1. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  2. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  3. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  4. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
  5. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  6. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  7. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 232
  8. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  9. 2013Exploiting Similarities among Languages for Machine Translationcited by 191
  10. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  11. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  12. 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
  13. 2017Learning Discrete Representations via Information Maximizing Self-Augmented Trainingcited by 149
  14. 2020Adversarial Training for Large Neural Language Modelscited by 111
  15. 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
  16. 2020CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLPcited by 79
  17. 2020Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpuscited by 49
  18. 2019Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsingcited by 46
  19. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  20. 2019XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answeringcited by 35
  21. 2020FILTER: An Enhanced Fusion Method for Cross-lingual Language Understandingcited by 29
  22. 2018Unsupervised Cross-lingual Transfer of Word Embedding Spacescited by 16
  23. 2019Cross-Lingual Dependency Parsing Using Code-Mixed TreeBankcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 17; 2024: 8; 2025: 4; 2026: 1.

41 papers.

Sort:

By area:

  1. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  2. 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
  3. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
  4. 2023Revisiting Machine Translation for Cross-lingual Classificationcited by 41
  5. 2023Consistency Analysis of ChatGPTcited by 37
  6. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  7. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  8. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  9. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  10. 2023Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?cited by 19
  11. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Modelscited by 13
  12. 2022CQR-SQL: Conversational Question Reformulation Enhanced Context-Dependent Text-to-SQL Parserscited by 11
29 more
  1. 2022ConNER: Consistency Training for Cross-lingual Named Entity Recognitioncited by 11
  2. 2023Enhancing Cross-lingual Transfer via Phonemic Transcription Integrationcited by 11
  3. 2023Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Studycited by 8
  4. 2023Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizercited by 7
  5. 2022Synergy with Translation Artifacts for Training and Inference in Multilingual Taskscited by 6
  6. 2024Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languagescited by 6
  7. 2021MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translationcited by 5
  8. 2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspectivecited by 5
  9. 2024Analysis of Multi-Source Language Training in Cross-Lingual Transfercited by 5
  10. 2024Enhancing Cross-Modal Fine-Tuning with Gradually Intermediate Modality Generationcited by 5
  11. 2021VAE based Text Style Transfer with Pivot Words Enhancement Learningcited by 4
  12. 2023XFEVER: Exploring Fact Verification across Languagescited by 4
  13. 2024CORI: CJKV Benchmark with Romanization Integration -- A step towards Cross-lingual Transfer Beyond Textual Scriptscited by 4
  14. 2024Improving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switchingcited by 3
  15. 2023HIT-SCIR at MMNLU-22: Consistency Regularization for Multilingual Spoken Language Understandingcited by 2
  16. 2023Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fastcited by 2
  17. 2023Interpreting Indirect Answers to Yes-No Questions in Multiple Languagescited by 2
  18. 2023Zero-Shot Cross-Lingual Sentiment Classification under Distribution Shift: an Exploratory Studycited by 2
  19. 2025Group Ligands Docking to Protein Pocketscited by 2
  20. 2023CoLaDa: A Collaborative Label Denoising Framework for Cross-lingual Named Entity Recognitioncited by 1
  21. 2024Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfercited by 1
  22. 2026Rethinking Cross-Modal Fine-Tuning: Optimizing the Interaction Between Feature Alignment and Target Fittingcited by 1
  23. 2021Cross-Lingual Language Model Meta-Pretrainingcited by 0
  24. 2023Test-Time Adaptation with Perturbation Consistency Learningcited by 0
  25. 2023Measuring Cross-Lingual Transferability of Multilingual Transformers on Sentence Classificationcited by 0
  26. 2023CUNI Submission to MRL 2023 Shared Task on Multi-lingual Multi-task Information Retrievalcited by 0
  27. 2023Sharing, Teaching and Aligning: Knowledgeable Transfer Learning for Cross-Lingual Machine Reading Comprehensioncited by 0
  28. 2024Generalization Measures for Zero-Shot Cross-Lingual Transfercited by 0
  29. 2025Cross-Prompt Encoder for Low-Performing Languagescited by 0