# An End-to-End Neural Network for Image-to-Audio Transformation

arXiv: https://arxiv.org/abs/2303.06078 · HTML: https://mlsanity.com/paper/2303.06078/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 10 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [Robust Scene Text Recognition with Automatic Rectification](https://mlsanity.com/paper/1603.03915/) (2016)
- [End-to-End Adversarial Text-to-Speech](https://mlsanity.com/paper/2006.03575/) (2020)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [One TTS Alignment To Rule Them All](https://mlsanity.com/paper/2108.10447/) (2021)
- [PortaSpeech: Portable and High-Quality Generative Text-to-Speech](https://mlsanity.com/paper/2109.15166/) (2021)
- [Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling](https://mlsanity.com/paper/2103.14574/) (2021)
- [Aggregation Cross-Entropy for Sequence Recognition](https://mlsanity.com/paper/1904.08364/) (2019)
- [HBONet: Harmonious Bottleneck on Two Orthogonal Dimensions](https://mlsanity.com/paper/1908.03888/) (2019)
- [Revisiting Classification Perspective on Scene Text Recognition](https://mlsanity.com/paper/2102.10884/) (2021)

## Built on by (most cited first, 2 of 2)

- [Compression of end-to-end non-autoregressive image-to-speech system for low-resourced devices](https://mlsanity.com/paper/2312.00174/) (2023)
- [Braille-to-Speech Generator: Audio Generation Based on Joint Fine-Tuning of CLIP and Fastspeech2](https://mlsanity.com/paper/2407.14212/) (2024)

## Abstract (arXiv)

This paper describes an end-to-end (E2E) neural architecture for the audio rendering of small portions of display content on low resource personal computing devices. It is intended to address the problem of accessibility for vision-impaired or vision-distracted users at the hardware level. Neural image-to-text (ITT) and text-to-speech (TTS) approaches are reviewed and a new technique is introduced to efficiently integrate them in a way that is both efficient and back-propagate-able, leading to a non-autoregressive E2E image-to-speech (ITS) neural network that is efficient and trainable. Experimental results are presented showing that, compared with the non-E2E approach, the proposed E2E system is 29% faster and uses 19% fewer parameters with a 2% reduction in phone accuracy. A future direction to address accuracy is presented.