# An Energy-based Model for Word-level AutoCompletion in Computer-aided Translation

arXiv: https://arxiv.org/abs/2407.20083 · HTML: https://mlsanity.com/paper/2407.20083/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 58 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)
- [Adversarial Retriever-Ranker for dense text retrieval](https://mlsanity.com/paper/2110.03611/) (2021)
- [Solving Math Word Problems via Cooperative Reasoning induced Language Models](https://mlsanity.com/paper/2210.16257/) (2022)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [LAMBADA: Backward Chaining for Automated Reasoning in Natural Language](https://mlsanity.com/paper/2212.13894/) (2022)
- [Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models](https://mlsanity.com/paper/2009.13267/) (2020)
- [Neural Machine Translation with Supervised Attention](https://mlsanity.com/paper/1609.04186/) (2016)
- [Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering](https://mlsanity.com/paper/1711.05116/) (2017)
- [Negative Sampling for Contrastive Representation Learning: A Review](https://mlsanity.com/paper/2206.00212/) (2022)
- [TranSmart: A Practical Interactive Machine Translation System](https://mlsanity.com/paper/2105.13072/) (2021)
- [Specialist or Generalist? Instruction Tuning for Specific NLP Tasks](https://mlsanity.com/paper/2310.15326/) (2023)
- [Question Answering as Programming for Solving Time-Sensitive Questions](https://mlsanity.com/paper/2305.14221/) (2023)
- [Universal Conditional Masked Language Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2203.09210/) (2022)
- [IntelliCAT: Intelligent Machine Translation Post-Editing with Quality Estimation and Translation Suggestion](https://mlsanity.com/paper/2105.12172/) (2021)
- [WeTS: A Benchmark for Translation Suggestion](https://mlsanity.com/paper/2110.05151/) (2021)
- [Exploring and Adapting Chinese GPT to Pinyin Input Method](https://mlsanity.com/paper/2203.00249/) (2022)
- [GWLAN: General Word-Level AutocompletioN for Computer-Aided Translation](https://mlsanity.com/paper/2105.14913/) (2021)
- [NewsDialogues: Towards Proactive News Grounded Conversation](https://mlsanity.com/paper/2308.06501/) (2023)
- [AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models](https://mlsanity.com/paper/2308.06507/) (2023)

## Abstract (arXiv)

Word-level AutoCompletion(WLAC) is a rewarding yet challenging task in Computer-aided Translation. Existing work addresses this task through a classification model based on a neural network that maps the hidden vector of the input context into its corresponding label (i.e., the candidate target word is treated as a label). Since the context hidden vector itself does not take the label into account and it is projected to the label through a linear classifier, the model can not sufficiently leverage valuable information from the source sentence as verified in our experiments, which eventually hinders its overall performance. To alleviate this issue, this work proposes an energy-based model for WLAC, which enables the context hidden vector to capture crucial information from the source sentence. Unfortunately, training and inference suffer from efficiency and effectiveness challenges, thereby we employ three simple yet effective strategies to put our model into practice. Experiments on four standard benchmarks demonstrate that our reranking-based approach achieves substantial improvements (about 6.07%) over the previous state-of-the-art model. Further analyses show that each strategy of our approach contributes to the final performance.