mlsanityData to 7 Oct 2026

ELMI: Interactive and Intelligent Sign Language Translation of Lyrics for Song Signing

arXiv 2409.09760 ↗First posted 15 Sep 2024cs.HC cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

d/Deaf and hearing song-signers have become prevalent across video-sharing platforms, but translating songs into sign language remains cumbersome and inaccessible. Our formative study revealed the challenges song-signers face, including semantic, syntactic, expressive, and rhythmic considerations in translations. We present ELMI, an accessible song-signing tool that assists in translating lyrics into sign language. ELMI enables users to edit glosses line-by-line, with real-time synced lyric and music video snippets. Users can also chat with a large language model-driven AI to discuss meaning, glossing, emoting, and timing. Through an exploratory study with 13 song-signers, we examined how ELMI facilitates their workflows and how song-signers leverage and receive an LLM-driven chat for translation. Participants successfully adopted ELMI to song-signing, with active discussions throughout. They also reported improved confidence and independence in their translations, finding ELMI encouraging, constructive, and informative. We discuss research and design implications for accessible and culturally sensitive song-signing translation tools.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 11 of its 84 references that are in the map. First posted on arXiv 15 Sep 2024.

Builds on

11 of 84 references.

References · 11

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  3. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 217
  4. 2018MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Languagecited by 118
  5. 2024LLMs are Good Sign Language Translatorscited by 76
  6. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  7. 2021Data Augmentation for Sign Language Gloss Translationcited by 28
  8. 2022Considerations for meaningful sign language machine translation based on glossescited by 24
  9. 2024Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendascited by 19
  10. 2022Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentationcited by 3
  11. 2017Break it Down for Me: A Study in Automated Lyric Annotationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.