# Modeling Order in Neural Word Embeddings at Scale

arXiv: https://arxiv.org/abs/1506.02338 · HTML: https://mlsanity.com/paper/1506.02338/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2015.

## Built on by (most cited first, 6 of 6)

- [Integrating Physiological Time Series and Clinical Notes with Transformer for Early Prediction of Sepsis](https://mlsanity.com/paper/2203.14469/) (2022)
- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [LIFE: Learning Individual Features for Multivariate Time Series Prediction with Missing Values](https://mlsanity.com/paper/2109.14844/) (2021)
- [Combining data assimilation and machine learning to estimate parameters of a convective-scale model](https://mlsanity.com/paper/2109.02953/) (2021)
- [MATNet: Multi-Level Fusion Transformer-Based Model for Day-Ahead PV Generation Forecasting](https://mlsanity.com/paper/2306.10356/) (2023)
- [AI Model Utilization Measurements For Finding Class Encoding Patterns](https://mlsanity.com/paper/2212.06576/) (2022)

## Abstract (arXiv)

Natural Language Processing (NLP) systems commonly leverage bag-of-words co-occurrence techniques to capture semantic and syntactic word relationships. The resulting word-level distributed representations often ignore morphological information, though character-level embeddings have proven valuable to NLP tasks. We propose a new neural language model incorporating both word order and character order in its embedding. The model produces several vector spaces with meaningful substructure, as evidenced by its performance of 85.8% on a recent word-analogy task, exceeding best published syntactic word-analogy scores by a 58% error margin. Furthermore, the model includes several parallel training methods, most notably allowing a skip-gram network with 160 billion parameters to be trained overnight on 3 multi-core CPUs, 14x larger than the previous largest neural network.