# Contextual LSTM (CLSTM) models for Large scale NLP tasks

arXiv: https://arxiv.org/abs/1602.06291 · HTML: https://mlsanity.com/paper/1602.06291/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2016.

## Built on by (most cited first, 15 of 15)

- [Focus Attention: Promoting Faithfulness and Diversity in Summarization](https://mlsanity.com/paper/2105.11921/) (2021)
- [Commonsense Reasoning for Conversational AI: A Survey of the State of the Art](https://mlsanity.com/paper/2302.07926/) (2023)
- [Self-Gated Memory Recurrent Network for Efficient Scalable HDR Deghosting](https://mlsanity.com/paper/2112.13050/) (2021)
- [AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimization](https://mlsanity.com/paper/2402.11940/) (2024)
- [Seeing Further and Wider: Joint Spatio-Temporal Enlargement for Micro-Video Popularity Prediction](https://mlsanity.com/paper/2604.20311/) (2026)
- [Attention-based Contextual Language Model Adaptation for Speech Recognition](https://mlsanity.com/paper/2106.01451/) (2021)
- [Deep Learning Models to Study Sentence Comprehension in the Human Brain](https://mlsanity.com/paper/2301.06340/) (2023)
- [Estimating Fund-Raising Performance for Start-up Projects from a Market Graph Perspective](https://mlsanity.com/paper/2105.12918/) (2021)
- [A Novel Stochastic LSTM Model Inspired by Quantum Machine Learning](https://mlsanity.com/paper/2305.10212/) (2023)
- [Privacy-Preserving Learning of Human Activity Predictors in Smart Environments](https://mlsanity.com/paper/2101.06564/) (2021)
- [Context-Aware Transformer Pre-Training for Answer Sentence Selection](https://mlsanity.com/paper/2305.15358/) (2023)
- [Generative AI for Internet of Things Security: Challenges and Opportunities](https://mlsanity.com/paper/2502.08886/) (2025)
- [OmniTrend: Content-Context Modeling for Scalable Social Popularity Prediction](https://mlsanity.com/paper/2604.26252/) (2026)
- [Structural Self-Supervised Objectives for Transformers](https://mlsanity.com/paper/2309.08272/) (2023)
- [Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs](https://mlsanity.com/paper/2401.16638/) (2024)

## Abstract (arXiv)

Documents exhibit sequential structure at multiple levels of abstraction (e.g., sentences, paragraphs, sections). These abstractions constitute a natural hierarchy for representing the context in which to infer the meaning of words and larger fragments of text. In this paper, we present CLSTM (Contextual LSTM), an extension of the recurrent neural network LSTM (Long-Short Term Memory) model, where we incorporate contextual features (e.g., topics) into the model. We evaluate CLSTM on three specific NLP tasks: word prediction, next sentence selection, and sentence topic prediction. Results from experiments run on two corpora, English documents in Wikipedia and a subset of articles from a recent snapshot of English Google News, indicate that using both words and topics as features improves performance of the CLSTM models over baseline LSTM models for these tasks. For example on the next sentence selection task, we get relative accuracy improvements of 21% for the Wikipedia dataset and 18% for the Google News dataset. This clearly demonstrates the significant benefit of using context appropriately in natural language (NL) tasks. This has implications for a wide variety of NL applications like question answering, sentence completion, paraphrase generation, and next utterance prediction in dialog systems.