mlsanityData to 14 Oct 2026

Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research

arXiv 2110.08975 ↗First posted 18 Oct 2021cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

AI is widely thought to be poised to transform business, yet current perceptions of the scope of this transformation may be myopic. Recent progress in natural language processing involving transformer language models (TLMs) offers a potential avenue for AI-driven business and societal transformation that is beyond the scope of what most currently foresee. We review this recent progress as well as recent literature utilizing text mining in top IS journals to develop an outline for how future IS research can benefit from these new techniques. Our review of existing IS literature reveals that suboptimal text mining techniques are prevalent and that the more advanced TLMs could be applied to enhance and increase IS research involving text data, and to enable new IS research topics, thus creating more value for the research community. This is possible because these techniques make it easier to develop very powerful custom systems and their performance is superior to existing methods for a wide range of tasks and applications. Further, multilingual language models make possible higher quality text analytics for research in multiple languages. We also identify new avenues for IS research, like language user interfaces, that may offer even greater potential for future IS research.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 90 of its 214 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

90 of 214 references.

References · 90

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  13. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  14. 2020Reformer: The Efficient Transformercited by 1,668
  15. 2020Rethinking Attention with Performerscited by 1,601
  16. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  17. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  18. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  19. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  20. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  21. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  22. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  23. 2020Recipes for building an open-domain chatbotcited by 593
  24. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  25. 2019Text Summarization with Pretrained Encoderscited by 556
  26. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  27. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  28. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  29. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  30. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  31. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  32. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  33. 2020Experience Grounds Languagecited by 242
  34. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  35. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  36. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  37. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  38. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1; 2025: 2.

7 papers.

  1. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
  2. 2023Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalizationcited by 28
  3. 2024Useful Blunders: Can Automated Speech Recognition Errors Improve Downstream Dementia Classification?cited by 9
  4. 2023WikiMuTe: A web-sourced dataset of semantic descriptions for music audiocited by 3
  5. 2025Investigating Tax Evasion Emergence Using Dual Large Language Model and Deep Reinforcement Learning Powered Agent-based Simulationcited by 3
  6. 2023Self-Admitted Technical Debt Detection Approaches: A Decade Systematic Reviewcited by 2
  7. 2025User-Based Sequential Modeling with Transformer Encoders for Insider Threat Detectioncited by 1