# Learning When to Concentrate or Divert Attention: Self-Adaptive Attention Temperature for Neural Machine Translation

arXiv: https://arxiv.org/abs/1808.07374 · HTML: https://mlsanity.com/paper/1808.07374/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2018.

## Built on by (most cited first, 10 of 10)

- [One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion](https://mlsanity.com/paper/2409.06366/) (2024)
- [Wait-info Policy: Balancing Source and Target at Information Level for Simultaneous Machine Translation](https://mlsanity.com/paper/2210.11220/) (2022)
- [Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulation](https://mlsanity.com/paper/2502.16872/) (2025)
- [PLADIS: Pushing the Limits of Attention in Diffusion Models at Inference Time by Leveraging Sparsity](https://mlsanity.com/paper/2503.07677/) (2025)
- [Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education](https://mlsanity.com/paper/2309.13059/) (2023)
- [Learning to Scale Temperature in Masked Self-Attention for Image Inpainting](https://mlsanity.com/paper/2302.06130/) (2023)
- [Optimal Attention Temperature Improves the Robustness of In-Context Learning under Distribution Shift in High Dimensions](https://mlsanity.com/paper/2511.01292/) (2025)
- [An empirical analysis of phrase-based and neural machine translation](https://mlsanity.com/paper/2103.03108/) (2021)
- [Synthesize, Partition, then Adapt: Eliciting Diverse Samples from Foundation Models](https://mlsanity.com/paper/2411.06722/) (2024)
- [CoANeRV: Coordinate-Aware Token-Space Neural Video Representation](https://mlsanity.com/paper/2608.13938/) (2026)

## Abstract (arXiv)

Most of the Neural Machine Translation (NMT) models are based on the sequence-to-sequence (Seq2Seq) model with an encoder-decoder framework equipped with the attention mechanism. However, the conventional attention mechanism treats the decoding at each time step equally with the same matrix, which is problematic since the softness of the attention for different types of words (e.g. content words and function words) should differ. Therefore, we propose a new model with a mechanism called Self-Adaptive Control of Temperature (SACT) to control the softness of attention by means of an attention temperature. Experimental results on the Chinese-English translation and English-Vietnamese translation demonstrate that our model outperforms the baseline models, and the analysis and the case study show that our model can attend to the most relevant elements in the source-side contexts and generate the translation of high quality.