# Positional Encoding to Control Output Sequence Length

arXiv: https://arxiv.org/abs/1904.07418 · HTML: https://mlsanity.com/paper/1904.07418/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2019.

## Built on by (most cited first, 30 of 56)

- [Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost](https://mlsanity.com/paper/2407.19825/) (2024)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [Transformers in Time-series Analysis: A Tutorial](https://mlsanity.com/paper/2205.01138/) (2022)
- [Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents](https://mlsanity.com/paper/2402.00798/) (2024)
- [SHAPE: Shifted Absolute Position Embedding for Transformers](https://mlsanity.com/paper/2109.05644/) (2021)
- [Controllable Abstractive Dialogue Summarization with Sketch Supervision](https://mlsanity.com/paper/2105.14064/) (2021)
- [Benchmarking Large Language Models on Controllable Generation under Diversified Instructions](https://mlsanity.com/paper/2401.00690/) (2024)
- [Uni6D: A Unified CNN Framework without Projection Breakdown for 6D Pose Estimation](https://mlsanity.com/paper/2203.14531/) (2022)
- [VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing](https://mlsanity.com/paper/2211.16934/) (2022)
- [Prompt-Based Length Controlled Generation with Multiple Control Types](https://mlsanity.com/paper/2406.10278/) (2024)
- [Machine Translation Verbosity Control for Automatic Dubbing](https://mlsanity.com/paper/2110.03847/) (2021)
- [Isometric MT: Neural Machine Translation for Automatic Dubbing](https://mlsanity.com/paper/2112.08682/) (2021)
- [Controllable Text Summarization: Unraveling Challenges, Approaches, and Prospects -- A Survey](https://mlsanity.com/paper/2311.09212/) (2023)
- [Rethinking Perturbations in Encoder-Decoders for Fast Training](https://mlsanity.com/paper/2104.01853/) (2021)
- [B2T Connection: Serving Stability and Performance in Deep Transformers](https://mlsanity.com/paper/2206.00330/) (2022)
- [LenAtten: An Effective Length Controlling Unit For Text Summarization](https://mlsanity.com/paper/2106.00316/) (2021)
- [Isochrony-Aware Neural Machine Translation for Automatic Dubbing](https://mlsanity.com/paper/2112.08548/) (2021)
- [Prompt-Based Length Controlled Generation with Reinforcement Learning](https://mlsanity.com/paper/2308.12030/) (2023)
- [A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarization](https://mlsanity.com/paper/2205.14522/) (2022)
- [Lost in Interpreting: Speech Translation from Source or Interpreter?](https://mlsanity.com/paper/2106.09343/) (2021)
- [Embedding Compression with Hashing for Efficient Representation Learning in Large-Scale Graph](https://mlsanity.com/paper/2208.05648/) (2022)
- [Length Controlled Generation for Black-box LLMs](https://mlsanity.com/paper/2412.14656/) (2024)
- [Factorizing Content and Budget Decisions in Abstractive Summarization of Long Documents](https://mlsanity.com/paper/2205.12486/) (2022)
- [Neural Class Expression Synthesis](https://mlsanity.com/paper/2111.08486/) (2021)
- [Unifying Cross-lingual Summarization and Machine Translation with Compression Rate](https://mlsanity.com/paper/2110.07936/) (2021)
- [Knowledge of Pretrained Language Models on Surface Information of Tokens](https://mlsanity.com/paper/2402.09808/) (2024)
- [Can LLMs Track Their Output Length? A Dynamic Feedback Mechanism for Precise Length Regulation](https://mlsanity.com/paper/2601.01768/) (2026)
- [Summarization with Precise Length Control](https://mlsanity.com/paper/2305.05171/) (2023)
- [InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models](https://mlsanity.com/paper/2406.11097/) (2024)
- [Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation](https://mlsanity.com/paper/2107.13689/) (2021)

## Abstract (arXiv)

Neural encoder-decoder models have been successful in natural language generation tasks. However, real applications of abstractive summarization must consider additional constraint that a generated summary should not exceed a desired length. In this paper, we propose a simple but effective extension of a sinusoidal positional encoding (Vaswani et al., 2017) to enable neural encoder-decoder model to preserves the length constraint. Unlike in previous studies where that learn embeddings representing each length, the proposed method can generate a text of any length even if the target length is not present in training data. The experimental results show that the proposed method can not only control the generation length but also improve the ROUGE scores.