# Pointing the Unknown Words

arXiv: https://arxiv.org/abs/1603.08148 · HTML: https://mlsanity.com/paper/1603.08148/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2016.

## Built on by (most cited first, 30 of 43)

- [Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld](https://mlsanity.com/paper/2311.16714/) (2023)
- [RevCore: Review-augmented Conversational Recommendation](https://mlsanity.com/paper/2106.00957/) (2021)
- [Data-to-text Generation with Macro Planning](https://mlsanity.com/paper/2102.02723/) (2021)
- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [PreSTU: Pre-Training for Scene-Text Understanding](https://mlsanity.com/paper/2209.05534/) (2022)
- [Learning Neural Templates for Recommender Dialogue System](https://mlsanity.com/paper/2109.12302/) (2021)
- [Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2108.02859/) (2021)
- [NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist](https://mlsanity.com/paper/2305.08566/) (2023)
- [History-Aware Hierarchical Transformer for Multi-session Open-domain Dialogue System](https://mlsanity.com/paper/2302.00907/) (2023)
- [RecInDial: A Unified Framework for Conversational Recommendation with Pretrained Language Models](https://mlsanity.com/paper/2110.07477/) (2021)
- [Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems](https://mlsanity.com/paper/2203.10610/) (2022)
- [Innovations in Neural Data-to-text Generation: A Survey](https://mlsanity.com/paper/2207.12571/) (2022)
- [Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap](https://mlsanity.com/paper/2109.11256/) (2021)
- [Commonsense and Named Entity Aware Knowledge Grounded Dialogue Generation](https://mlsanity.com/paper/2205.13928/) (2022)
- [Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function](https://mlsanity.com/paper/2303.13797/) (2023)
- [Revisiting Negation in Neural Machine Translation](https://mlsanity.com/paper/2107.12203/) (2021)
- [Iterative Decoding for Compositional Generalization in Transformers](https://mlsanity.com/paper/2110.04169/) (2021)
- [AutoTemplate: A Simple Recipe for Lexically Constrained Text Generation](https://mlsanity.com/paper/2211.08387/) (2022)
- [HistAlign: Improving Context Dependency in Language Generation by Aligning with History](https://mlsanity.com/paper/2305.04782/) (2023)
- [A Template-guided Hybrid Pointer Network for Knowledge-basedTask-oriented Dialogue Systems](https://mlsanity.com/paper/2106.05830/) (2021)
- [EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLM](https://mlsanity.com/paper/2505.19905/) (2025)
- [DICTDIS: Dictionary Constrained Disambiguation for Improved NMT](https://mlsanity.com/paper/2210.06996/) (2022)
- [Towards a Unified Conversational Recommendation System: Multi-task Learning via Contextualized Knowledge Distillation](https://mlsanity.com/paper/2310.18119/) (2023)
- [OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMs](https://mlsanity.com/paper/2503.11858/) (2025)
- [May the Force Be with Your Copy Mechanism: Enhanced Supervised-Copy Method for Natural Language Generation](https://mlsanity.com/paper/2112.10360/) (2021)
- [How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?](https://mlsanity.com/paper/2208.05225/) (2022)
- [Discourse Representation Structure Parsing for Chinese](https://mlsanity.com/paper/2306.09725/) (2023)
- [Towards Accurate Translation via Semantically Appropriate Application of Lexical Constraints](https://mlsanity.com/paper/2306.12089/) (2023)
- [Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach](https://mlsanity.com/paper/2409.09009/) (2024)
- [BioCopy: A Plug-And-Play Span Copy Mechanism in Seq2Seq Models](https://mlsanity.com/paper/2109.12533/) (2021)

## Abstract (arXiv)

The problem of rare and unknown words is an important issue that can potentially influence the performance of many NLP systems, including both the traditional count-based and the deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context.~We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known.~We observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset using our proposed model.