mlsanityData to 7 Oct 2026

Distributed Representations of Words and Phrases and their Compositionality

arXiv 1310.4546 ↗First posted 16 Oct 2013cs.CL cs.LG stat.MLcited by 3,462

Not on the map. Why?

Cited by 45 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.

Key facts. Cited by 3,462 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Oct 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 1; 2019: 1; 2020: 3; 2021: 1047; 2022: 685; 2023: 616; 2024: 506; 2025: 391; 2026: 211.

3,462 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  10. 2023Steering Language Models With Activation Engineeringcited by 824
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
38 more
  1. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  2. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  3. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  4. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2021Image Segmentation Using Text and Image Promptscited by 509
  7. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  8. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  9. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  10. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  11. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  12. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  13. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  14. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  15. 2023Language Models Represent Space and Timecited by 306
  16. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  17. 2022UL2: Unifying Language Learning Paradigmscited by 292
  18. 2023Function Vectors in Large Language Modelscited by 292
  19. 2021Graph Self-Supervised Learning: A Surveycited by 289
  20. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  21. 2021How to Train Your Energy-Based Modelscited by 271
  22. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  23. 2021Training Spiking Neural Networks Using Lessons From Deep Learningcited by 251
  24. 2023A Cookbook of Self-Supervised Learningcited by 248
  25. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  26. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  27. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  28. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  29. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  30. 2024Generative Representational Instruction Tuningcited by 227
  31. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  32. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  33. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  34. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  35. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  36. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  37. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
  38. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198

50 most cited of 3,462 All 3,462 (JSON)