# Easy Come, Easy Go? Examining the Perceptions and Learning Effects of LLM-based Chatbot in the Context of Search-as-Learning

arXiv: https://arxiv.org/abs/2410.01396 · HTML: https://mlsanity.com/paper/2410.01396/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 75 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 8 of 8)

- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models](https://mlsanity.com/paper/2402.11723/) (2024)
- [AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation](https://mlsanity.com/paper/2402.14978/) (2024)
- [A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education](https://mlsanity.com/paper/2312.03173/) (2023)
- [AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails](https://mlsanity.com/paper/2402.09216/) (2024)
- [CoAIcoder: Examining the Effectiveness of AI-assisted Human-to-Human Collaboration in Qualitative Analysis](https://mlsanity.com/paper/2304.05560/) (2023)
- [GPTutor: a ChatGPT-powered programming tool for code explanation](https://mlsanity.com/paper/2305.01863/) (2023)
- [How Teachers Can Use Large Language Models and Bloom's Taxonomy to Create Educational Quizzes](https://mlsanity.com/paper/2401.05914/) (2024)

## Built on by (most cited first, 1 of 1)

- [From Superficial Outputs to Superficial Learning: Risks of Large Language Models in Education](https://mlsanity.com/paper/2509.21972/) (2025)

## Abstract (arXiv)

The cognitive process of Search-as-Learning (SAL) is most effective when searching promotes active encoding of information. The rise of LLMs-based chatbots, which provide instant answers, introduces a trade-off between efficiency and depth of processing. Such answer-centric approaches accelerate information access, but they also raise concerns about shallower learning. To examine these issues in the context of SAL, we conducted a large-scale survey of educators and students to capture perceived risks and benefits of LLM-based chatbots. In addition, we adopted the encoding-storage paradigm to design a within-subjects experiment, where participants (N=92) engaged in SAL tasks using three different modalities: books, search engines, and chatbots. Our findings provide a counterintuitive insight into stakeholder concerns: while LLM-based chatbots and search engines validated perceived benefits on learning efficiency by outperforming book-based search in immediate conceptual understanding, they did not result in a long-term inferiority as feared. Our study provides insights for designing human-AI collaborative learning systems that promote cognitive engagement by balancing learning efficiency and long-term knowledge retention.