mlsanityData to 7 Oct 2026

IMO: Greedy Layer-Wise Sparse Representation Learning for Out-of-Distribution Text Classification with Pre-trained Models

arXiv 2404.13504 ↗First posted 21 Apr 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Machine learning models have made incredible progress, but they still struggle when applied to examples from unseen domains. This study focuses on a specific problem of domain generalization, where a model is trained on one source domain and tested on multiple target domains that are unseen during training. We propose IMO: Invariant features Masks for Out-of-Distribution text classification, to achieve OOD generalization by learning invariant features. During training, IMO would learn sparse mask layers to remove irrelevant features for prediction, where the remaining features keep invariant. Additionally, IMO has an attention module at the token level to focus on tokens that are useful for prediction. Our comprehensive experiments show that IMO substantially outperforms strong baselines in terms of various evaluation metrics and settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 76 references that are in the map. First posted on arXiv 21 Apr 2024.

Builds on

49 of 76 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2019Invariant Risk Minimizationcited by 1,832
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  11. 2020In Search of Lost Domain Generalizationcited by 1,042
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
37 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  4. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  5. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  6. 2013Domain Generalization via Invariant Feature Representationcited by 622
  7. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  8. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  9. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  10. 2020Learning to Learn Single Domain Generalizationcited by 295
  11. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  12. 2021Learning to Diversify for Single Domain Generalizationcited by 170
  13. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  14. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  15. 2022Causality Inspired Representation Learning for Domain Generalizationcited by 136
  16. 2020Learning to Balance Specificity and Invariance for In and Out of Domain Generalizationcited by 128
  17. 2018Invariance, Causality and Robustnesscited by 123
  18. 2020Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustnesscited by 121
  19. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  20. 2021Towards Principled Disentanglement for Domain Generalizationcited by 91
  21. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  22. 2021Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Testscited by 77
  23. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  24. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 66
  25. 2022MaskTune: Mitigating Spurious Correlations by Forcing to Explorecited by 66
  26. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  27. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  28. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  29. 2022Improving Multi-Task Generalization via Regularizing Spurious Correlationcited by 28
  30. 2022Domain Generalization by Learning and Removing Domain-specific Featurescited by 28
  31. 2019Domain Adaptive Text Style Transfercited by 25
  32. 2022Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustnesscited by 23
  33. 2019Accurate and Compact Convolutional Neural Networks with Trained Binarizationcited by 20
  34. 2024RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversationscited by 14
  35. 2023DRCFS: Doubly Robust Causal Feature Selectioncited by 12
  36. 2023Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discoverycited by 10
  37. 2022Paraphrasing Techniques for Maritime QA systemcited by 7

Built on by

Not cited yet.