# Sign Language Translation with Iterative Prototype

arXiv: https://arxiv.org/abs/2308.12191 · HTML: https://mlsanity.com/paper/2308.12191/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Progressive Image Deraining Networks: A Better and Simpler Baseline](https://mlsanity.com/paper/1901.09221/) (2019)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Two-Stream Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2211.01367/) (2022)
- [Deep Snake for Real-Time Instance Segmentation](https://mlsanity.com/paper/2001.01629/) (2020)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [Fast Interactive Object Annotation with Curve-GCN](https://mlsanity.com/paper/1903.06874/) (2019)
- [Video-based Sign Language Recognition without Temporal Segmentation](https://mlsanity.com/paper/1801.10111/) (2018)
- [Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective](https://mlsanity.com/paper/1908.08597/) (2019)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation](https://mlsanity.com/paper/2203.04287/) (2022)
- [TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation](https://mlsanity.com/paper/2010.05468/) (2020)
- [Multi-channel Transformers for Multi-articulatory Sign Language Translation](https://mlsanity.com/paper/2009.00299/) (2020)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation](https://mlsanity.com/paper/2203.04074/) (2022)
- [DocScanner: Robust Document Image Rectification with Progressive Learning](https://mlsanity.com/paper/2110.14968/) (2021)
- [Revisiting Local Descriptor for Improved Few-Shot Classification](https://mlsanity.com/paper/2103.16009/) (2021)

## Built on by (most cited first, 20 of 20)

- [Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2405.04164/) (2024)
- [Improving Gloss-free Sign Language Translation by Reducing Representation Density](https://mlsanity.com/paper/2405.14312/) (2024)
- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator](https://mlsanity.com/paper/2411.17799/) (2024)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation](https://mlsanity.com/paper/2505.15438/) (2025)
- [Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2405.05672/) (2024)
- [Spotter+GPT: Turning Sign Spottings into Sentences with LLMs](https://mlsanity.com/paper/2403.10434/) (2024)
- [SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval](https://mlsanity.com/paper/2407.16394/) (2024)
- [Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2503.06484/) (2025)
- [MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation](https://mlsanity.com/paper/2601.19577/) (2026)
- [Text-Driven Diffusion Model for Sign Language Production](https://mlsanity.com/paper/2503.15914/) (2025)
- [DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model](https://mlsanity.com/paper/2411.17248/) (2024)
- [Contrastive Pretraining with Dual Visual Encoders for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2507.10306/) (2025)
- [SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation](https://mlsanity.com/paper/2606.28626/) (2026)
- [SSL-SSAW: Self-Supervised Learning with Sigmoid Self-Attention Weighting for Question-Based Sign Language Translation](https://mlsanity.com/paper/2509.14036/) (2025)
- [Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standards](https://mlsanity.com/paper/2606.19352/) (2026)
- [DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2607.27614/) (2026)
- [ODE-Based Transformer Decoders for Iterative Sign Language Translation](https://mlsanity.com/paper/2608.11352/) (2026)
- [Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation](https://mlsanity.com/paper/2609.07965/) (2026)

## Abstract (arXiv)

This paper presents IP-SLT, a simple yet effective framework for sign language translation (SLT). Our IP-SLT adopts a recurrent structure and enhances the semantic representation (prototype) of the input sign language video via an iterative refinement manner. Our idea mimics the behavior of human reading, where a sentence can be digested repeatedly, till reaching accurate understanding. Technically, IP-SLT consists of feature extraction, prototype initialization, and iterative prototype refinement. The initialization module generates the initial prototype based on the visual feature extracted by the feature extraction module. Then, the iterative refinement module leverages the cross-attention mechanism to polish the previous prototype by aggregating it with the original video feature. Through repeated refinement, the prototype finally converges to a more stable and accurate state, leading to a fluent and appropriate translation. In addition, to leverage the sequential dependence of prototypes, we further propose an iterative distillation loss to compress the knowledge of the final iteration into previous ones. As the autoregressive decoding process is executed only once in inference, our IP-SLT is ready to improve various SLT systems with acceptable overhead. Extensive experiments are conducted on public benchmarks to demonstrate the effectiveness of the IP-SLT.