# Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment arXiv: https://arxiv.org/abs/2307.03744 ยท HTML: https://mlsanity.com/paper/2307.03744/ ## Key facts - Cited by 36 papers in the map (data through 14 Oct 2026). - First posted on arXiv 7 Jul 2023. ## Built on by (most cited first, 30 of 36) - [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023) - [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024) - ["I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust](https://mlsanity.com/paper/2405.00623/) (2024) - [eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data](https://mlsanity.com/paper/2402.08831/) (2024) - [Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking](https://mlsanity.com/paper/2410.03703/) (2024) - [Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies](https://mlsanity.com/paper/2502.08554/) (2025) - [JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models](https://mlsanity.com/paper/2406.09321/) (2024) - [The Impossibility of Fair LLMs](https://mlsanity.com/paper/2406.03198/) (2024) - [Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search](https://mlsanity.com/paper/2502.04951/) (2025) - [One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations](https://mlsanity.com/paper/2405.05581/) (2024) - [StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization](https://mlsanity.com/paper/2504.05804/) (2025) - [Detecting Generated Native Ads in Conversational Search](https://mlsanity.com/paper/2402.04889/) (2024) - [News Source Citing Patterns in AI Search Systems](https://mlsanity.com/paper/2507.05301/) (2025) - [ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models](https://mlsanity.com/paper/2505.13176/) (2025) - [Co-audit: tools to help humans double-check AI-generated content](https://mlsanity.com/paper/2310.01297/) (2023) - [An Assessment of Model-On-Model Deception](https://mlsanity.com/paper/2405.12999/) (2024) - [Cost-Efficient Prompt Engineering for Unsupervised Entity Resolution](https://mlsanity.com/paper/2310.06174/) (2023) - [Large Language Model Agents for Improving Engagement with Behavior Change Interventions: Application to Digital Mindfulness](https://mlsanity.com/paper/2407.13067/) (2024) - [NExT-Search: Rebuilding User Feedback Ecosystem for Generative AI Search](https://mlsanity.com/paper/2505.14680/) (2025) - [Lossless Token Sequence Compression via Meta-Tokens](https://mlsanity.com/paper/2506.00307/) (2025) - [AmazonQAC: A Large-Scale, Naturalistic Query Autocomplete Dataset](https://mlsanity.com/paper/2411.04129/) (2024) - [Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agents](https://mlsanity.com/paper/2502.15690/) (2024) - [Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learning](https://mlsanity.com/paper/2505.18831/) (2025) - [Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition](https://mlsanity.com/paper/2603.18557/) (2026) - [Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queries](https://mlsanity.com/paper/2408.08401/) (2024) - [Hermit Kingdom Through the Lens of Multiple Perspectives: A Case Study of LLM Hallucination on North Korea](https://mlsanity.com/paper/2501.05981/) (2025) - [Agentic Search Engine for Real-Time IoT Data](https://mlsanity.com/paper/2503.12255/) (2025) - [AI Revolution on Chat Bot: Evidence from a Randomized Controlled Experiment](https://mlsanity.com/paper/2401.10956/) (2024) - [Aligning Agents like Large Language Models](https://mlsanity.com/paper/2406.04208/) (2024) - [Generative AI's aggregated knowledge versus web-based curated knowledge](https://mlsanity.com/paper/2410.12091/) (2024) ## Abstract (arXiv) Recent advances in the development of large language models are rapidly changing how online applications function. LLM-based search tools, for instance, offer a natural language interface that can accommodate complex queries and provide detailed, direct responses. At the same time, there have been concerns about the veracity of the information provided by LLM-based tools due to potential mistakes or fabrications that can arise in algorithmically generated text. In a set of online experiments we investigate how LLM-based search changes people's behavior relative to traditional search, and what can be done to mitigate overreliance on LLM-based output. Participants in our experiments were asked to solve a series of decision tasks that involved researching and comparing different products, and were randomly assigned to do so with either an LLM-based search tool or a traditional search engine. In our first experiment, we find that participants using the LLM-based tool were able to complete their tasks more quickly, using fewer but more complex queries than those who used traditional search. Moreover, these participants reported a more satisfying experience with the LLM-based search tool. When the information presented by the LLM was reliable, participants using the tool made decisions with a comparable level of accuracy to those using traditional search, however we observed overreliance on incorrect information when the LLM erred. Our second experiment further investigated this issue by randomly assigning some users to see a simple color-coded highlighting scheme to alert them to potentially incorrect or misleading information in the LLM responses. Overall we find that this confidence-based highlighting substantially increases the rate at which users spot incorrect information, improving the accuracy of their overall decisions while leaving most other measures unaffected.