# The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive Visualization

arXiv: https://arxiv.org/abs/2311.13057 · HTML: https://mlsanity.com/paper/2311.13057/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2023.

## Built on by (most cited first, 24 of 24)

- [Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents](https://mlsanity.com/paper/2502.16069/) (2025)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process](https://mlsanity.com/paper/2504.12488/) (2025)
- [Collage is the New Writing: Exploring the Fragmentation of Text and User Interfaces in AI Tools](https://mlsanity.com/paper/2405.17217/) (2024)
- [LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology](https://mlsanity.com/paper/2509.13978/) (2025)
- [Where Do I 'Add the Egg'?: Exploring Agency and Ownership in AI Creative Co-Writing Systems](https://mlsanity.com/paper/2509.15440/) (2025)
- [Constella: Supporting Storywriters' Interconnected Character Creation through LLM-based Multi-Agents](https://mlsanity.com/paper/2507.05820/) (2025)
- [RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.06056/) (2025)
- [Narrix: Remixing Narrative Strategies from Examples for Story Writing](https://mlsanity.com/paper/2604.07643/) (2026)
- [AI Rivalry as a Craft: How Resisting and Embracing Generative AI Reshape Writing Professions](https://mlsanity.com/paper/2503.09901/) (2025)
- [Attribution Gradients: Incrementally Unfolding Citations for Critical Examination of Attributed AI Answers](https://mlsanity.com/paper/2510.00361/) (2025)
- [XAgen: An Explainability Tool for Identifying and Correcting Failures in Multi-Agent Workflows](https://mlsanity.com/paper/2512.17896/) (2025)
- [The AI Memory Gap: Users Misremember What They Created With AI or Without](https://mlsanity.com/paper/2509.11851/) (2025)
- [Investigating Writing Professionals' Relationships with Generative AI: How Combined Perceptions of Rivalry and Collaboration Shape Work Practices and Outcomes](https://mlsanity.com/paper/2602.08227/) (2026)
- [AI in the Writing Process: How Purposeful AI Support Fosters Student Writing](https://mlsanity.com/paper/2506.20595/) (2025)
- [Who Owns the Text? Design Patterns for Preserving Authorship in AI-Assisted Writing](https://mlsanity.com/paper/2601.10236/) (2026)
- [What happens when reviewers receive AI feedback in their reviews?](https://mlsanity.com/paper/2602.13817/) (2026)
- [Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research](https://mlsanity.com/paper/2604.17843/) (2026)
- [Mixed-Initiative Context: Structuring and Managing Context for Human-AI Collaboration](https://mlsanity.com/paper/2604.07121/) (2026)
- [Human-AI Interaction Traces as Blackout Poetry: Reframing AI-Supported Writing as Found-Text Creativity](https://mlsanity.com/paper/2604.09605/) (2026)
- [FocalLens: Visualizing Narratives through Focalization](https://mlsanity.com/paper/2604.14456/) (2026)
- [When AI Blurs the Boundaries of Contribution: An Empirical Study of Authorship Calibration](https://mlsanity.com/paper/2607.15006/) (2026)
- [AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysis](https://mlsanity.com/paper/2608.17834/) (2026)
- [Likelihood Is Not Authorship: Can We Reliably Measure Human Contribution in AI-Assisted Writing?](https://mlsanity.com/paper/2610.19767/) (2026)

## Abstract (arXiv)

The use of Large Language Models (LLMs) for writing has sparked controversy both among readers and writers. On one hand, writers are concerned that LLMs will deprive them of agency and ownership, and readers are concerned about spending their time on text generated by soulless machines. On the other hand, AI-assistance can improve writing as long as writers can conform to publisher policies, and as long as readers can be assured that a text has been verified by a human. We argue that a system that captures the provenance of interaction with an LLM can help writers retain their agency, conform to policies, and communicate their use of AI to publishers and readers transparently. Thus we propose HaLLMark, a tool for visualizing the writer's interaction with the LLM. We evaluated HaLLMark with 13 creative writers, and found that it helped them retain a sense of control and ownership of the text.