# Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval

arXiv: https://arxiv.org/abs/1502.06922 · HTML: https://mlsanity.com/paper/1502.06922/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2015.

## Built on by (most cited first, 30 of 54)

- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Neural ranking models for document retrieval](https://mlsanity.com/paper/2102.11903/) (2021)
- [Code Search based on Context-aware Code Translation](https://mlsanity.com/paper/2202.08029/) (2022)
- [Attentive Deep Neural Networks for Legal Document Retrieval](https://mlsanity.com/paper/2212.13899/) (2022)
- [Learning Job Titles Similarity from Noisy Skill Labels](https://mlsanity.com/paper/2207.00494/) (2022)
- [Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal](https://mlsanity.com/paper/2212.13023/) (2022)
- [Query Rewriting via Cycle-Consistent Translation for E-Commerce Search](https://mlsanity.com/paper/2103.00800/) (2021)
- [Graph-based Multilingual Product Retrieval in E-commerce Search](https://mlsanity.com/paper/2105.02978/) (2021)
- [Divide and Conquer: Text Semantic Matching with Disentangled Keywords and Intents](https://mlsanity.com/paper/2203.02898/) (2022)
- [Training-free Neural Architecture Search for RNNs and Transformers](https://mlsanity.com/paper/2306.00288/) (2023)
- [Deep Semantic Graph Learning via LLM based Node Enhancement](https://mlsanity.com/paper/2502.07982/) (2025)
- [RankDNN: Learning to Rank for Few-shot Learning](https://mlsanity.com/paper/2211.15320/) (2022)
- [Learning to Ask: Conversational Product Search via Representation Learning](https://mlsanity.com/paper/2411.14466/) (2024)
- [Glitter or Gold? Deriving Structured Insights from Sustainability Reports via Large Language Models](https://mlsanity.com/paper/2310.05628/) (2023)
- [Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings](https://mlsanity.com/paper/2403.14001/) (2024)
- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [Match-Ignition: Plugging PageRank into Transformer for Long-form Text Matching](https://mlsanity.com/paper/2101.06423/) (2021)
- [Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation](https://mlsanity.com/paper/2203.07687/) (2022)
- [Debiasing Neural Retrieval via In-batch Balancing Regularization](https://mlsanity.com/paper/2205.09240/) (2022)
- [Attention-based Dynamic Multilayer Graph Neural Networks for Loan Default Prediction](https://mlsanity.com/paper/2402.00299/) (2024)
- [Unsupervised Sentence-embeddings by Manifold Approximation and Projection](https://mlsanity.com/paper/2102.03795/) (2021)
- [Improving Text Semantic Similarity Modeling through a 3D Siamese Network](https://mlsanity.com/paper/2307.09274/) (2023)
- [Deep Bag-of-Words Model: An Efficient and Interpretable Relevance Architecture for Chinese E-Commerce](https://mlsanity.com/paper/2407.09395/) (2024)
- [Explainable LLM-driven Multi-dimensional Distillation for E-Commerce Relevance Learning](https://mlsanity.com/paper/2411.13045/) (2024)
- [Deep Natural Language Processing for LinkedIn Search](https://mlsanity.com/paper/2108.13300/) (2021)
- [Bank transactions embeddings help to uncover current macroeconomics](https://mlsanity.com/paper/2110.12000/) (2021)
- [A Survey on Awesome Korean NLP Datasets](https://mlsanity.com/paper/2112.01624/) (2021)
- [Robust Interaction-Based Relevance Modeling for Online e-Commerce Search](https://mlsanity.com/paper/2406.02135/) (2024)
- [Centrality-aware Product Retrieval and Ranking](https://mlsanity.com/paper/2410.15930/) (2024)

## Abstract (arXiv)

This paper develops a model that addresses sentence embedding, a hot topic in current natural language processing research, using recurrent neural networks with Long Short-Term Memory (LSTM) cells. Due to its ability to capture long term memory, the LSTM-RNN accumulates increasingly richer information as it goes through the sentence, and when it reaches the last word, the hidden layer of the network provides a semantic representation of the whole sentence. In this paper, the LSTM-RNN is trained in a weakly supervised manner on user click-through data logged by a commercial web search engine. Visualization and analysis are performed to understand how the embedding process works. The model is found to automatically attenuate the unimportant words and detects the salient keywords in the sentence. Furthermore, these detected keywords are found to automatically activate different cells of the LSTM-RNN, where words belonging to a similar topic activate the same cell. As a semantic representation of the sentence, the embedding vector can be used in many different applications. These automatic keyword detection and topic allocation abilities enabled by the LSTM-RNN allow the network to perform document retrieval, a difficult language processing task, where the similarity between the query and documents can be measured by the distance between their corresponding sentence embedding vectors computed by the LSTM-RNN. On a web search task, the LSTM-RNN embedding is shown to significantly outperform several existing state of the art methods. We emphasize that the proposed model generates sentence embedding vectors that are specially useful for web document retrieval tasks. A comparison with a well known general sentence embedding method, the Paragraph Vector, is performed. The results show that the proposed method in this paper significantly outperforms it for web document retrieval task.