# End-to-End real time tracking of children's reading with pointer network

arXiv: https://arxiv.org/abs/2310.11486 · HTML: https://mlsanity.com/paper/2310.11486/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 21 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 6 of 6)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020)
- [Towards Real-time Mispronunciation Detection in Kids' Speech](https://mlsanity.com/paper/2003.01765/) (2020)
- [End-to-end acoustic modelling for phone recognition of young readers](https://mlsanity.com/paper/2103.02899/) (2021)

## Built on by (most cited first, 2 of 2)

- [In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions](https://mlsanity.com/paper/2604.22817/) (2026)
- [X2-NativeCursor: Native-Token Text Progress Tracking for Incremental-Text Streaming Codec TTS](https://mlsanity.com/paper/2609.09677/) (2026)

## Abstract (arXiv)

In this work, we explore how a real time reading tracker can be built efficiently for children's voices. While previously proposed reading trackers focused on ASR-based cascaded approaches, we propose a fully end-to-end model making it less prone to lags in voice tracking. We employ a pointer network that directly learns to predict positions in the ground truth text conditioned on the streaming speech. To train this pointer network, we generate ground truth training signals by using forced alignment between the read speech and the text being read on the training set. Exploring different forced alignment models, we find a neural attention based model is at least as close in alignment accuracy to the Montreal Forced Aligner, but surprisingly is a better training signal for the pointer network. Our results are reported on one adult speech data (TIMIT) and two children's speech datasets (CMU Kids and Reading Races). Our best model can accurately track adult speech with 87.8% accuracy and the much harder and disfluent children's speech with 77.1% accuracy on CMU Kids data and a 65.3% accuracy on the Reading Races dataset.