# Linguistics-Vision Monotonic Consistent Network for Sign Language Production

arXiv: https://arxiv.org/abs/2412.16944 · HTML: https://mlsanity.com/paper/2412.16944/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 22 Dec 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Progressive Transformers for End-to-End Sign Language Production](https://mlsanity.com/paper/2004.14874/) (2020)
- [Learning Two-Branch Neural Networks for Image-Text Matching Tasks](https://mlsanity.com/paper/1704.03470/) (2017)
- [Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregation](https://mlsanity.com/paper/2303.11579/) (2023)
- [G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model](https://mlsanity.com/paper/2208.09141/) (2022)
- [Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment](https://mlsanity.com/paper/2312.15645/) (2023)
- [Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production](https://mlsanity.com/paper/2412.13609/) (2024)
- [Dual-path Collaborative Generation Network for Emotional Video Captioning](https://mlsanity.com/paper/2408.03006/) (2024)
- [Including Facial Expressions in Contextual Embeddings for Sign Language Generation](https://mlsanity.com/paper/2202.05383/) (2022)
- [Unified Instance and Knowledge Alignment Pretraining for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2110.13398/) (2021)
- [A Token-level Contrastive Framework for Sign Language Translation](https://mlsanity.com/paper/2204.04916/) (2022)
- [Modality Alignment Meets Federated Broadcasting](https://mlsanity.com/paper/2411.15837/) (2024)

## Built on by (most cited first, 5 of 5)

- [SignAligner: Harmonizing Complementary Pose Modalities for Coherent Sign Language Generation](https://mlsanity.com/paper/2506.11621/) (2025)
- [Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering](https://mlsanity.com/paper/2508.02362/) (2025)
- [Motion is the Choreographer: Learning Latent Pose Dynamics for Seamless Sign Language Generation](https://mlsanity.com/paper/2508.04049/) (2025)
- [Towards Multi-View Sign Language Understanding: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2604.18184/) (2026)
- [Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standards](https://mlsanity.com/paper/2606.19352/) (2026)

## Abstract (arXiv)

Sign Language Production (SLP) aims to generate sign videos corresponding to spoken language sentences, where the conversion of sign Glosses to Poses (G2P) is the key step. Due to the cross-modal semantic gap and the lack of word-action correspondence labels for strong supervision alignment, the SLP suffers huge challenges in linguistics-vision consistency. In this work, we propose a Transformer-based Linguistics-Vision Monotonic Consistent Network (LVMCN) for SLP, which constrains fine-grained cross-modal monotonic alignment and coarse-grained multimodal semantic consistency in language-visual cues through Cross-modal Semantic Aligner (CSA) and Multimodal Semantic Comparator (MSC). In the CSA, we constrain the implicit alignment between corresponding gloss and pose sequences by computing the cosine similarity association matrix between cross-modal feature sequences (i.e., the order consistency of fine-grained sign glosses and actions). As for MSC, we construct multimodal triplets based on paired and unpaired samples in batch data. By pulling closer the corresponding text-visual pairs and pushing apart the non-corresponding text-visual pairs, we constrain the semantic co-occurrence degree between corresponding gloss and pose sequences (i.e., the semantic consistency of coarse-grained textual sentences and sign videos). Extensive experiments on the popular PHOENIX14T benchmark show that the LVMCN outperforms the state-of-the-art.