# SeeSay: An Assistive Device for the Visually Impaired Using Retrieval Augmented Generation

arXiv: https://arxiv.org/abs/2410.03771 · HTML: https://mlsanity.com/paper/2410.03771/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 1 of its 6 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 1 of 1)

- [Deep Learning based Wearable Assistive System for Visually Impaired People](https://mlsanity.com/paper/1908.03364/) (2019)

## Abstract (arXiv)

In this paper, we present SeeSay, an assistive device designed for individuals with visual impairments. This system leverages large language models (LLMs) for speech recognition and visual querying. It effectively identifies, records, and responds to the user's environment by providing audio guidance using retrieval-augmented generation (RAG). Our experiments demonstrate the system's capability to recognize its surroundings and respond to queries with audio feedback in diverse settings. We hope that the SeeSay system will facilitate users' comprehension and recollection of their surroundings, thereby enhancing their environmental perception, improving navigational capabilities, and boosting overall independence.