# ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles

arXiv: https://arxiv.org/abs/2310.15428 · HTML: https://mlsanity.com/paper/2310.15428/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 63 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory](https://mlsanity.com/paper/1704.01074/) (2017)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots](https://mlsanity.com/paper/1612.01627/) (2016)
- [Sparks: Inspiration for Science Writing using Language Models](https://mlsanity.com/paper/2110.07640/) (2021)
- [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://mlsanity.com/paper/2208.07852/) (2022)
- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- ["What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models](https://mlsanity.com/paper/2304.06597/) (2023)
- [RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions](https://mlsanity.com/paper/2302.09466/) (2023)
- [3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows](https://mlsanity.com/paper/2210.11603/) (2022)
- [Opal: Multimodal Image Generation for News Illustration](https://mlsanity.com/paper/2204.09007/) (2022)
- [On Training Instance Selection for Few-Shot Neural Text Generation](https://mlsanity.com/paper/2107.03176/) (2021)
- [PopBlends: Strategies for Conceptual Blending with Large Language Models](https://mlsanity.com/paper/2111.04920/) (2021)
- [ScatterShot: Interactive In-context Example Curation for Text Transformation](https://mlsanity.com/paper/2302.07346/) (2023)
- [PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models](https://mlsanity.com/paper/2304.01964/) (2023)
- [Multi-modal Synthesis of Regular Expressions](https://mlsanity.com/paper/1908.03316/) (2019)

## Built on by (most cited first, 30 of 33)

- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles](https://mlsanity.com/paper/2407.00870/) (2024)
- [Interactive Debugging and Steering of Multi-Agent AI Systems](https://mlsanity.com/paper/2503.02068/) (2025)
- [Inverse Constitutional AI: Compressing Preferences into Principles](https://mlsanity.com/paper/2406.06560/) (2024)
- [TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students](https://mlsanity.com/paper/2410.04078/) (2024)
- [What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use](https://mlsanity.com/paper/2409.08775/) (2024)
- [Designing for Human-Agent Alignment: Understanding what humans want from their agents](https://mlsanity.com/paper/2404.04289/) (2024)
- [HonestLLM: Toward an Honest and Helpful Large Language Model](https://mlsanity.com/paper/2406.00380/) (2024)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Social Skill Training with Large Language Models](https://mlsanity.com/paper/2404.04204/) (2024)
- [Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflow](https://mlsanity.com/paper/2402.09894/) (2024)
- [C3AI: Crafting and Evaluating Constitutions for Constitutional AI](https://mlsanity.com/paper/2502.15861/) (2025)
- [Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products](https://mlsanity.com/paper/2410.12071/) (2024)
- [ConstitutionalExperts: Training a Mixture of Principle-based Prompts](https://mlsanity.com/paper/2403.04894/) (2024)
- [Self-Directed Synthetic Dialogues and Revisions Technical Report](https://mlsanity.com/paper/2407.18421/) (2024)
- [Interactive Prompt Debugging with Sequence Salience](https://mlsanity.com/paper/2404.07498/) (2024)
- [Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors](https://mlsanity.com/paper/2409.18203/) (2024)
- [SPRI: Aligning Large Language Models with Context-Situated Principles](https://mlsanity.com/paper/2502.03397/) (2025)
- [Survey of User Interface Design and Interaction Techniques in Generative AI Applications](https://mlsanity.com/paper/2410.22370/) (2024)
- [Human-AI Complementarity: A Goal for Amplified Oversight](https://mlsanity.com/paper/2510.26518/) (2025)
- [FeedQUAC: Quick Unobtrusive AI-Generated Commentary](https://mlsanity.com/paper/2504.16416/) (2025)
- [Interpreting and Controlling Model Behavior via Constitutions for Atomic Concept Edits](https://mlsanity.com/paper/2602.00092/) (2026)
- [Contextual Confidence and Generative AI](https://mlsanity.com/paper/2311.01193/) (2023)
- [POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models](https://mlsanity.com/paper/2406.03843/) (2024)
- [Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning](https://mlsanity.com/paper/2501.15727/) (2025)
- [DoubleAgents: Human-Agent Alignment in a Socially Embedded Workflow](https://mlsanity.com/paper/2509.12626/) (2025)
- [Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations](https://mlsanity.com/paper/2509.25492/) (2025)
- [Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behavior](https://mlsanity.com/paper/2510.12728/) (2025)
- [LLM Prompt Evaluation for Educational Applications](https://mlsanity.com/paper/2601.16134/) (2026)
- [Generative Experiences for Digital Mental Health Interventions: Evidence from a Randomized Study](https://mlsanity.com/paper/2604.07558/) (2026)

## Abstract (arXiv)

Large language model (LLM) prompting is a promising new approach for users to create and customize their own chatbots. However, current methods for steering a chatbot's outputs, such as prompt engineering and fine-tuning, do not support users in converting their natural feedback on the model's outputs to changes in the prompt or model. In this work, we explore how to enable users to interactively refine model outputs through their feedback, by helping them convert their feedback into a set of principles (i.e. a constitution) that dictate the model's behavior. From a formative study, we (1) found that users needed support converting their feedback into principles for the chatbot and (2) classified the different principle types desired by users. Inspired by these findings, we developed ConstitutionMaker, an interactive tool for converting user feedback into principles, to steer LLM-based chatbots. With ConstitutionMaker, users can provide either positive or negative feedback in natural language, select auto-generated feedback, or rewrite the chatbot's response; each mode of feedback automatically generates a principle that is inserted into the chatbot's prompt. In a user study with 14 participants, we compare ConstitutionMaker to an ablated version, where users write their own principles. With ConstitutionMaker, participants felt that their principles could better guide the chatbot, that they could more easily convert their feedback into principles, and that they could write principles more efficiently, with less mental demand. ConstitutionMaker helped users identify ways to improve the chatbot, formulate their intuitive responses to the model into feedback, and convert this feedback into specific and clear principles. Together, these findings inform future tools that support the interactive critiquing of LLM outputs.