mlsanityData to 7 Oct 2026

Deep contextualized word representations

arXiv 1802.05365 ↗First posted 15 Feb 2018cs.CLcited by 2,285

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

Key facts. Cited by 2,285 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 1; 2020: 3; 2021: 859; 2022: 557; 2023: 392; 2024: 234; 2025: 152; 2026: 85.

2,285 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
38 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  5. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  6. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  10. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  11. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  12. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  13. 2023A Comprehensive Overview of Large Language Modelscited by 639
  14. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  15. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  16. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  17. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  18. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  19. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  20. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  21. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  22. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  23. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  24. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  25. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  26. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  27. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  28. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  29. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  30. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  31. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  32. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  33. 2022Confident Adaptive Language Modelingcited by 297
  34. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  35. 2022UL2: Unifying Language Learning Paradigmscited by 292
  36. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  37. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  38. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274

50 most cited of 2,285 All 2,285 (JSON)