# Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review

arXiv: https://arxiv.org/abs/2501.12557 · HTML: https://mlsanity.com/paper/2501.12557/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 99 of its 196 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 30 of 99)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [Mapping the Increasing Use of LLMs in Scientific Papers](https://mlsanity.com/paper/2404.01268/) (2024)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM](https://mlsanity.com/paper/2404.12259/) (2024)
- [A Design Space for Intelligent and Interactive Writing Assistants](https://mlsanity.com/paper/2403.14117/) (2024)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces](https://mlsanity.com/paper/2309.14459/) (2023)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers](https://mlsanity.com/paper/2101.09157/) (2021)
- ["It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents](https://mlsanity.com/paper/2309.11653/) (2023)
- [A Bibliometric Review of Large Language Models Research from 2017 to 2023](https://mlsanity.com/paper/2304.02020/) (2023)
- [Rethinking Model Evaluation as Narrowing the Socio-Technical Gap](https://mlsanity.com/paper/2306.03100/) (2023)

## Built on by (most cited first, 16 of 16)

- [CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experiments](https://mlsanity.com/paper/2509.13588/) (2025)
- [VTutor: An Open-Source SDK for Generative AI-Powered Animated Pedagogical Agents with Multi-Media Output](https://mlsanity.com/paper/2502.04103/) (2025)
- [Vibe Check: Understanding the Effects of LLM-Based Conversational Agents' Personality and Alignment on User Perceptions in Goal-Oriented Tasks](https://mlsanity.com/paper/2509.09870/) (2025)
- [Prompt Engineer: Analyzing Hard and Soft Skill Requirements in the AI Job Market](https://mlsanity.com/paper/2506.00058/) (2025)
- [Reflexis: Supporting Reflexivity and Rigor in Collaborative Qualitative Analysis through Design for Deliberation](https://mlsanity.com/paper/2601.15445/) (2026)
- [Plans They Abandon, Reports They Author: The Narrative Layer of Autonomous Agents](https://mlsanity.com/paper/2609.12205/) (2026)
- [When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applications](https://mlsanity.com/paper/2602.00044/) (2026)
- [A Generative AI System for Biomedical Data Discovery with Grammar-Based Visualizations](https://mlsanity.com/paper/2509.16454/) (2025)
- [YAC: Bridging Natural Language and Interactive Visual Exploration with Generative AI for Biomedical Data Discovery](https://mlsanity.com/paper/2509.19182/) (2025)
- [Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research](https://mlsanity.com/paper/2604.17843/) (2026)
- [Implicature in Interaction: Understanding Implicature Improves Alignment in Human-LLM Interaction](https://mlsanity.com/paper/2510.25426/) (2025)
- [PersonaCite: VoC-Grounded Interviewable Agentic Synthetic AI Personas for Verifiable User and Design Research](https://mlsanity.com/paper/2601.22288/) (2026)
- [What Do We Mean by 'Pilot Study': Early Findings from a Meta-Review of Pilot Study Reporting at CHI](https://mlsanity.com/paper/2602.13488/) (2026)
- [Thinking in Graphs with CoMAP: A Shared Visual Workspace for Designing Project-Based Learning](https://mlsanity.com/paper/2604.06200/) (2026)
- [LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions](https://mlsanity.com/paper/2609.14849/) (2026)
- [Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows](https://mlsanity.com/paper/2609.27327/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been positioned to revolutionize HCI, by reshaping not only the interfaces, design patterns, and sociotechnical systems that we study, but also the research practices we use. To-date, however, there has been little understanding of LLMs' uptake in HCI. We address this gap via a systematic literature review of 153 CHI papers from 2020-24 that engage with LLMs. We taxonomize: (1) domains where LLMs are applied; (2) roles of LLMs in HCI projects; (3) contribution types; and (4) acknowledged limitations and risks. We find LLM work in 10 diverse domains, primarily via empirical and artifact contributions. Authors use LLMs in five distinct roles, including as research tools or simulated users. Still, authors often raise validity and reproducibility concerns, and overwhelmingly study closed models. We outline opportunities to improve HCI research with and on LLMs, and provide guiding questions for researchers to consider the validity and appropriateness of LLM-related work.