# Non-Parametric Online Learning from Human Feedback for Neural Machine Translation

arXiv: https://arxiv.org/abs/2109.11136 · HTML: https://mlsanity.com/paper/2109.11136/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2010.00710/) (2020)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Guiding Neural Machine Translation with Retrieved Translation Pieces](https://mlsanity.com/paper/1804.02559/) (2018)
- [Neural Machine Translation with Monolingual Translation Memory](https://mlsanity.com/paper/2105.11269/) (2021)
- [Adaptive Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.13022/) (2021)
- [Learning to Remember Translation History with a Continuous Cache](https://mlsanity.com/paper/1711.09367/) (2017)
- [Non-Parametric Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1903.00058/) (2019)
- [Joint Training for Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1803.00353/) (2018)
- [Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/2109.06604/) (2021)
- [Online Learning for Effort Reduction in Interactive Neural Machine Translation](https://mlsanity.com/paper/1802.03594/) (2018)
- [Correct-and-Memorize: Learning to Translate from Interactive Revisions](https://mlsanity.com/paper/1907.03468/) (2019)
- [A User-Study on Online Adaptation of Neural Machine Translation to Human Post-Edits](https://mlsanity.com/paper/1712.04853/) (2017)
- [Learning to Reuse Translations: Guiding Neural Machine Translation with Examples](https://mlsanity.com/paper/1911.10732/) (2019)
- [Incremental Adaptation of NMT for Professional Post-editors: A User Study](https://mlsanity.com/paper/1906.08996/) (2019)
- [Demonstration of a Neural Machine Translation System with Online Learning for Translators](https://mlsanity.com/paper/1906.09000/) (2019)

## Built on by (most cited first, 8 of 8)

- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibration](https://mlsanity.com/paper/2212.02216/) (2022)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [Simple and Scalable Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12188/) (2023)
- [ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation](https://mlsanity.com/paper/2405.11912/) (2024)
- [Federated Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12211/) (2023)
- [Human-in-the-loop Machine Translation with Large Language Model](https://mlsanity.com/paper/2310.08908/) (2023)

## Abstract (arXiv)

We study the problem of online learning with human feedback in the human-in-the-loop machine translation, in which the human translators revise the machine-generated translations and then the corrected translations are used to improve the neural machine translation (NMT) system. However, previous methods require online model updating or additional translation memory networks to achieve high-quality performance, making them inflexible and inefficient in practice. In this paper, we propose a novel non-parametric online learning method without changing the model structure. This approach introduces two k-nearest-neighbor (knn) modules: one module memorizes the human feedback, which is the correct sentences provided by human translators, while the other balances the usage of the history human feedback and original NMT models adaptively. Experiments conducted on EMEA and JRC-Acquis benchmarks demonstrate that our proposed method obtains substantial improvements on translation accuracy and achieves better adaptation performance with less repeating human correction operations.