mlsanityData to 7 Oct 2026

SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation

arXiv 2609.21709 ↗First posted 18 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) provide limited support for sign language interaction. Unifying sign language translation (SLT) and generation (SLG) to enable sign language as both input and output can reduce switching between separate models during sign-text interaction. We present SignGPT, a unified, pose-based framework for gloss-free SLT and SLG. SignGPT integrates part-aware hierarchical representations of body, hand, and facial motion into a shared language model and employs asymmetric multi-token prediction and progressive training for bidirectional modeling. We evaluate SignGPT on How2Sign (ASL) and Phoenix-2014T (DGS) through benchmark comparisons, qualitative analyses, and component ablations. An exploratory study with 12 Deaf ASL signers assesses an LLM-mediated sign-to-sign response pipeline, highlighting the potential of unified modeling to support sign language conversation (SLC).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 60 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

36 of 60 references.

References · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  4. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  6. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  7. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  8. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  9. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  10. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  11. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  12. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 172
24 more
  1. 2024LLMs are Good Sign Language Translatorscited by 76
  2. 2020TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translationcited by 75
  3. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
  4. 2023SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understandingcited by 59
  5. 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 58
  6. 2025Uni-Sign: Toward Unified Sign Language Understanding at Scalecited by 55
  7. 2023Neural Sign Actors: A diffusion model for 3D sign language production from textcited by 48
  8. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  9. 2024MotionChain: Conversational Motion Controllers via Multimodal Promptscited by 30
  10. 2024T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Textcited by 30
  11. 2023SignDiff: Diffusion Model for American Sign Language Productioncited by 25
  12. 2025Lost in Translation, Found in Context: Sign Language Translation with Contextual Cuescited by 25
  13. 2024Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observationcited by 16
  14. 2024Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Productioncited by 16
  15. 2024SignAvatar: Sign Language 3D Motion Reconstruction and Generationcited by 14
  16. 2025Towards AI-driven Sign Language Generation with Non-manual Markerscited by 13
  17. 2024FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluationcited by 10
  18. 2025Multilingual Gloss-free Sign Language Translation: Towards Building a Sign Language Foundation Modelcited by 7
  19. 2025SiLVERScore: Semantically-Aware Embeddings for Sign Language Generation Evaluationcited by 4
  20. 2024Fingerspelling within Sign Language Translationcited by 3
  21. 2026Reimagining Sign Language Technologies: Analyzing Translation Work of Chinese Deaf Online Content Creatorscited by 2
  22. 2026CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Languagecited by 2
  23. 2024ELMI: Interactive and Intelligent Sign Language Translation of Lyrics for Song Signingcited by 1
  24. 2025SIGNER: Temporally Grounded Sign Language Generation via Time-Resolved Conditioningcited by 1

Built on by

Not cited yet.