# ELMI: Interactive and Intelligent Sign Language Translation of Lyrics for Song Signing

arXiv: https://arxiv.org/abs/2409.09760 · HTML: https://mlsanity.com/paper/2409.09760/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 84 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [LLMs are Good Sign Language Translators](https://mlsanity.com/paper/2404.00925/) (2024)
- ["I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust](https://mlsanity.com/paper/2405.00623/) (2024)
- [Data Augmentation for Sign Language Gloss Translation](https://mlsanity.com/paper/2105.07476/) (2021)
- [Considerations for meaningful sign language machine translation based on glosses](https://mlsanity.com/paper/2211.15464/) (2022)
- [Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendas](https://mlsanity.com/paper/2403.02563/) (2024)
- [Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation](https://mlsanity.com/paper/2204.05953/) (2022)
- [Break it Down for Me: A Study in Automated Lyric Annotation](https://mlsanity.com/paper/1708.03492/) (2017)

## Built on by (most cited first, 1 of 1)

- [SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation](https://mlsanity.com/paper/2609.21709/) (2026)

## Abstract (arXiv)

d/Deaf and hearing song-signers have become prevalent across video-sharing platforms, but translating songs into sign language remains cumbersome and inaccessible. Our formative study revealed the challenges song-signers face, including semantic, syntactic, expressive, and rhythmic considerations in translations. We present ELMI, an accessible song-signing tool that assists in translating lyrics into sign language. ELMI enables users to edit glosses line-by-line, with real-time synced lyric and music video snippets. Users can also chat with a large language model-driven AI to discuss meaning, glossing, emoting, and timing. Through an exploratory study with 13 song-signers, we examined how ELMI facilitates their workflows and how song-signers leverage and receive an LLM-driven chat for translation. Participants successfully adopted ELMI to song-signing, with active discussions throughout. They also reported improved confidence and independence in their translations, finding ELMI encouraging, constructive, and informative. We discuss research and design implications for accessible and culturally sensitive song-signing translation tools.