# Multimodal Behavioral Patterns Analysis with Eye-Tracking and LLM-Based Reasoning

arXiv: https://arxiv.org/abs/2507.18252 · HTML: https://mlsanity.com/paper/2507.18252/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 50 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [Common 7B Language Models Already Possess Strong Math Capabilities](https://mlsanity.com/paper/2403.04706/) (2024)
- [Towards Interpretable Mental Health Analysis with Large Language Models](https://mlsanity.com/paper/2304.03347/) (2023)
- [Making Large Language Models Better Data Creators](https://mlsanity.com/paper/2310.20111/) (2023)
- [An LLM Agent for Automatic Geospatial Data Analysis](https://mlsanity.com/paper/2410.18792/) (2024)
- [Introduction to Eye Tracking: A Hands-On Tutorial for Students and Practitioners](https://mlsanity.com/paper/2404.15435/) (2024)

## Built on by (most cited first, 1 of 1)

- [Lazy or Efficient? Towards Accessible Eye-Tracking Event Detection Using LLMs](https://mlsanity.com/paper/2604.13243/) (2026)

## Abstract (arXiv)

Eye-tracking data reveals valuable insights into users' cognitive states but is difficult to analyze due to its structured, non-linguistic nature. While large language models (LLMs) excel at reasoning over text, they struggle with temporal and numerical data. This paper presents a multimodal human-AI collaborative framework designed to enhance cognitive pattern extraction from eye-tracking signals. The framework includes: (1) a multi-stage pipeline using horizontal and vertical segmentation alongside LLM reasoning to uncover latent gaze patterns; (2) an Expert-Model Co-Scoring Module that integrates expert judgment with LLM output to generate trust scores for behavioral interpretations; and (3) a hybrid anomaly detection module combining LSTM-based temporal modeling with LLM-driven semantic analysis. Our results across several LLMs and prompt strategies show improvements in consistency, interpretability, and performance, with up to 50% accuracy in difficulty prediction tasks. This approach offers a scalable, interpretable solution for cognitive modeling and has broad potential in adaptive learning, human-computer interaction, and educational analytics.