# ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience

arXiv: https://arxiv.org/abs/2307.01135 · HTML: https://mlsanity.com/paper/2307.01135/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 32 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 3 of 3)

- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://mlsanity.com/paper/2303.10130/) (2023)
- [Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models](https://mlsanity.com/paper/2304.07619/) (2023)

## Built on by (most cited first, 12 of 12)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow](https://mlsanity.com/paper/2307.07367/) (2023)
- [MegaFake: A Theory-Driven Dataset of Fake News Generated by Large Language Models](https://mlsanity.com/paper/2408.11871/) (2024)
- [Exploring ChatGPT for Next-generation Information Retrieval: Opportunities and Challenges](https://mlsanity.com/paper/2402.11203/) (2024)
- [Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queries](https://mlsanity.com/paper/2408.08401/) (2024)
- [From Clicks to Conversations: Evaluating the Effectiveness of Conversational Agents in Statistical Analysis](https://mlsanity.com/paper/2502.08114/) (2025)
- [When Content is Goliath and Algorithm is David: The Style and Semantic Effects of Generative Search Engine](https://mlsanity.com/paper/2509.14436/) (2025)
- [The Evolution of Learning: Assessing the Transformative Impact of Generative AI on Higher Education](https://mlsanity.com/paper/2404.10551/) (2024)
- [Understanding Critical Thinking in Generative Artificial Intelligence Use: Development, Validation, and Correlates of the Critical Thinking in AI Use Scale](https://mlsanity.com/paper/2512.12413/) (2025)
- [The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search](https://mlsanity.com/paper/2605.28498/) (2026)
- [Rethinking Search: A Study of University Students' Perspectives on Using LLMs and Traditional Search Engines in Academic Problem Solving](https://mlsanity.com/paper/2510.17726/) (2025)
- [Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment](https://mlsanity.com/paper/2610.07023/) (2026)

## Abstract (arXiv)

The advent of ChatGPT, a large language model-powered chatbot, has prompted questions about its potential implications for traditional search engines. In this study, we investigate the differences in user behavior when employing search engines and chatbot tools for information-seeking tasks. We carry out a randomized online experiment, dividing participants into two groups: one using a ChatGPT-like tool and the other using a Google Search-like tool. Our findings reveal that the ChatGPT group consistently spends less time on all tasks, with no significant difference in overall task performance between the groups. Notably, ChatGPT levels user search performance across different education levels and excels in answering straightforward questions and providing general solutions but falls short in fact-checking tasks. Users perceive ChatGPT's responses as having higher information quality compared to Google Search, despite displaying a similar level of trust in both tools. Furthermore, participants using ChatGPT report significantly better user experiences in terms of usefulness, enjoyment, and satisfaction, while perceived ease of use remains comparable between the two tools. However, ChatGPT may also lead to overreliance and generate or replicate misinformation, yielding inconsistent results. Our study offers valuable insights for search engine management and highlights opportunities for integrating chatbot technologies into search engine designs.