# A New Era in Human Factors Engineering: A Survey of the Applications and Prospects of Large Multimodal Models

arXiv: https://arxiv.org/abs/2405.13426 · HTML: https://mlsanity.com/paper/2405.13426/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 86 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [Generative artificial intelligence enhances creativity but reduces the diversity of novel content](https://mlsanity.com/paper/2312.00506/) (2023)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [Towards Mitigating Hallucination in Large Language Models via Self-Reflection](https://mlsanity.com/paper/2310.06271/) (2023)
- [Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions](https://mlsanity.com/paper/2312.02913/) (2023)
- [Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language Models](https://mlsanity.com/paper/2307.11991/) (2023)
- [Large Language Models as Zero-Shot Human Models for Human-Robot Interaction](https://mlsanity.com/paper/2303.03548/) (2023)
- [EEG-GPT: Exploring Capabilities of Large Language Models for EEG Classification and Interpretation](https://mlsanity.com/paper/2401.18006/) (2024)
- [LaMI: Large Language Models for Multi-Modal Human-Robot Interaction](https://mlsanity.com/paper/2401.15174/) (2024)
- [AccidentGPT: Accident Analysis and Prevention from V2X Environmental Perception with Multi-modal Large Model](https://mlsanity.com/paper/2312.13156/) (2023)
- [Large Language Models in Analyzing Crash Narratives -- A Comparative Study of ChatGPT, BARD and GPT-4](https://mlsanity.com/paper/2308.13563/) (2023)
- [A Compact and Interpretable Convolutional Neural Network for Cross-Subject Driver Drowsiness Detection from Single-Channel EEG](https://mlsanity.com/paper/2106.00613/) (2021)
- [AccidentGPT: Large Multi-Modal Foundation Model for Traffic Accident Analysis](https://mlsanity.com/paper/2401.03040/) (2024)
- [Enhancing Traffic Incident Management with Large Language Models: A Hybrid Machine Learning Approach for Severity Classification](https://mlsanity.com/paper/2403.13547/) (2024)
- [Generative AI for Product Design: Getting the Right Design and the Design Right](https://mlsanity.com/paper/2306.01217/) (2023)
- [Reading Users' Minds from What They Say: An Investigation into LLM-based Empathic Mental Inference](https://mlsanity.com/paper/2403.13301/) (2024)
- [Virtual Co-Pilot: Multimodal Large Language Model-enabled Quick-access Procedures for Single Pilot Operations](https://mlsanity.com/paper/2403.16645/) (2024)

## Built on by (most cited first, 5 of 5)

- [ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation](https://mlsanity.com/paper/2406.09961/) (2024)
- [DiLLS: Interactive Diagnosis of LLM-based Multi-agent Systems via Layered Summary of Agent Behaviors](https://mlsanity.com/paper/2602.05446/) (2026)
- [Personalize Your Large Vision-language Models With In-context Prompt Tuning](https://mlsanity.com/paper/2605.31513/) (2026)
- [A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environments](https://mlsanity.com/paper/2502.10419/) (2025)
- [InconLens: Interactive Visual Diagnosis of Behavioral Inconsistencies in LLM-based Agentic Systems](https://mlsanity.com/paper/2603.28106/) (2026)

## Abstract (arXiv)

In recent years, the potential applications of Large Multimodal Models (LMMs) in fields such as healthcare, social psychology, and industrial design have attracted wide research attention, providing new directions for human factors research. For instance, LMM-based smart systems have become novel research subjects of human factors studies, and LMM introduces new research paradigms and methodologies to this field. Therefore, this paper aims to explore the applications, challenges, and future prospects of LMM in the domain of human factors and ergonomics through an expert-LMM collaborated literature review. Specifically, a novel literature review method is proposed, and research studies of LMM-based accident analysis, human modelling and intervention design are introduced. Subsequently, the paper discusses future trends of the research paradigm and challenges of human factors and ergonomics studies in the era of LMMs. It is expected that this study can provide a valuable perspective and serve as a reference for integrating human factors with artificial intelligence.