mlsanityData to 14 Oct 2026

Co-audit: tools to help humans double-check AI-generated content

arXiv 2310.01297 ↗First posted 2 Oct 2023cs.HC cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Users are increasingly being warned to check AI-generated content for correctness. Still, as LLMs (and other generative models) generate more complex output, such as summaries, tables, or code, it becomes harder for the user to audit or evaluate the output for quality or correctness. Hence, we are seeing the emergence of tool-assisted experiences to help the user double-check a piece of AI-generated content. We refer to these as co-audit tools. Co-audit tools complement prompt engineering techniques: one helps the user construct the input prompt, while the other helps them check the output response. As a specific example, this paper describes recent research on co-audit tools for spreadsheet computations powered by generative models. We explain why co-audit experiences are essential for any application of generative AI where quality is important and errors are consequential (as is common in spreadsheet computations). We propose a preliminary list of principles for co-audit, and outline research challenges.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 45 of its 82 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

45 of 82 references.

References · 45

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  10. 2023BloombergGPT: A Large Language Model for Financecited by 939
  11. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  12. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
33 more
  1. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  2. 2018Actionable Recourse in Linear Classificationcited by 328
  3. 2023Using an LLM to Help With Code Understandingcited by 228
  4. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  5. 2023On the Reliability of Watermarks for Large Language Modelscited by 189
  6. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 124
  7. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
  8. 2019Efficient Search for Diverse Coherent Explanationscited by 111
  9. 2023InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuningcited by 82
  10. 2023Is GPT-4 a Good Data Analyst?cited by 70
  11. 2023Evaluating Language Models for Mathematics through Interactionscited by 70
  12. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  13. 2023nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Modelscited by 60
  14. 2023"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Modelscited by 48
  15. 2023Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Modelscited by 48
  16. 2023Supporting Human-AI Collaboration in Auditing LLMs with LLMscited by 46
  17. 2020A systematic review and taxonomy of explanations in decision support and recommender systemscited by 45
  18. 2022What is it like to program with artificial intelligence?cited by 39
  19. 2022Automatic Code Documentation Generation Using GPT-3cited by 37
  20. 2023Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experimentcited by 36
  21. 2023Mirages: On Anthropomorphism in Dialogue Systemscited by 30
  22. 2023FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysiscited by 28
  23. 2023Demonstration of InsightPilot: An LLM-Empowered Automated Data Exploration Systemcited by 20
  24. 2023What Should Data Science Education Do with Large Language Models?cited by 13
  25. 2023ProcessGPT: Transforming Business Process Management with Generative Artificial Intelligencecited by 11
  26. 2023Detecting ChatGPT: A Survey of the State of Detecting ChatGPT-Generated Textcited by 10
  27. 2022Is explainable AI a race against model complexity?cited by 9
  28. 2023Generation-based Code Review Automation: How Far Are We?cited by 9
  29. 2023Validating AI-Generated Code with Live Programmingcited by 8
  30. 2023ConceptEVA: Concept-Based Interactive Exploration and Customization of Document Summariescited by 7
  31. 2023Designing Interfaces for Human-Computer Communication: An On-Going Collection of Considerationscited by 3
  32. 2020Towards making formal methods normal: meeting developers where they arecited by 2
  33. 2023AutoScrum: Automating Project Planning Using Large Language Modelscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4.

5 papers.

  1. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  2. 2023Selenite: Scaffolding Online Sensemaking with Comprehensive Overviews Elicited from Large Language Modelscited by 10
  3. 2024Sociotechnical Implications of Generative Artificial Intelligence for Information Accesscited by 6
  4. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  5. 2024Data Analysis in the Era of Generative AIcited by 6