TorontoCL at CMCL 2021 Shared Task: RoBERTa with Multi-Stage Fine-Tuning for Eye-Tracking Prediction
Not on the map. Why?
Abstract (arXiv)
Eye movement data during reading is a useful source of information for understanding language comprehension processes. In this paper, we describe our submission to the CMCL 2021 shared task on predicting human reading patterns. Our model uses RoBERTa with a regression layer to predict 5 eye-tracking features. We train the model in two stages: we first fine-tune on the Provo corpus (another eye-tracking dataset), then fine-tune on the task data. We compare different Transformer models and apply ensembling methods to improve the performance. Our final submission achieves a MAE score of 3.929, ranking 3rd place out of 13 teams that participated in this shared task.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 4 of its 12 references that are in the map. First posted on arXiv 15 Apr 2021.
Builds on
4 of 12 references.
References · 4
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
- 2019Towards Making a Dependency Parser Seecited by 5
Built on by
Citing papers per year: 2022: 3; 2023: 0; 2024: 1; 2025: 3.
7 papers.
- 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 11
- 2025An Ensemble Classification Approach in A Multi-Layered Large Language Model Framework for Disease Predictioncited by 4
- 2022Zero Shot Crosslingual Eye-Tracking Data Prediction using Multilingual Transformer Modelscited by 3
- 2025OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responsescited by 3
- 2025Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directionscited by 2
- 2022UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentationcited by 1
- 2022Team 'UFAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Modelscited by 0
- 2025An Ensemble Classification Approach in A Multi-Layered Large Language Model Framework for Disease Prediction2 Sep 2025
- 2025Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions9 Apr 2025
- 2025OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses13 Mar 2025
- 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models2 Oct 2024
- 2022UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentation7 Oct 2022
- 2022Team 'UFAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Models11 Apr 2022
- 2022Zero Shot Crosslingual Eye-Tracking Data Prediction using Multilingual Transformer Models30 Mar 2022