mlsanityData to 14 Oct 2026

Multilingual Communication System with Deaf Individuals Utilizing Natural and Visual Languages

arXiv 2212.00305 ↗First posted 1 Dec 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

According to the World Federation of the Deaf, more than two hundred sign languages exist. Therefore, it is challenging to understand deaf individuals, even proficient sign language users, resulting in a barrier between the deaf community and the rest of society. To bridge this language barrier, we propose a novel multilingual communication system, namely MUGCAT, to improve the communication efficiency of sign language users. By converting recognized specific hand gestures into expressive pictures, which is universal usage and language independence, our MUGCAT system significantly helps deaf people convey their thoughts. To overcome the limitation of sign language usage, which is mostly impossible to translate into complete sentences for ordinary people, we propose to reconstruct meaningful sentences from the incomplete translation of sign language. We also measure the semantic similarity of generated sentences with fragmented recognized hand gestures to keep the original meaning. Experimental results show that the proposed system can work in a real-time manner and synthesize exquisite stunning illustrations and meaningful sentences from a few hand gestures of sign language. This proves that our MUGCAT has promising potential in assisting deaf communication.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

25 of 35 references.

References · 25

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  11. 2022On Distillation of Guided Diffusion Modelscited by 725
  12. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
13 more
  1. 2017Deep Mutual Learningcited by 628
  2. 2016Self-critical Sequence Training for Image Captioningcited by 547
  3. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  4. 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 275
  5. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 218
  6. 2020X-Linear Attention Networks for Image Captioningcited by 198
  7. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  8. 2021Visual Alignment Constraint for Continuous Sign Language Recognitioncited by 80
  9. 2022Deep Equilibrium Optical Flow Estimationcited by 64
  10. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  11. 2021Deep Two-View Structure-from-Motion Revisitedcited by 39
  12. 2021Monocular Depth Estimation through Virtual-world Supervision and Real-world SfM Self-Supervisioncited by 24
  13. 2020Mutual Modality Learning for Video Action Classificationcited by 3

Built on by

Not cited yet.