# Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning

arXiv: https://arxiv.org/abs/2501.15727 · HTML: https://mlsanity.com/paper/2501.15727/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 70 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 23 of 23)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models](https://mlsanity.com/paper/2310.20410/) (2023)
- [Towards artificial general intelligence via a multimodal foundation model](https://mlsanity.com/paper/2110.14378/) (2021)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- ["What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models](https://mlsanity.com/paper/2304.06597/) (2023)
- ["We Need Structured Output": Towards User-centered Constraints on Large Language Model Output](https://mlsanity.com/paper/2404.07362/) (2024)
- [ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles](https://mlsanity.com/paper/2310.15428/) (2023)
- [Prompt Problems: A New Programming Exercise for the Generative AI Era](https://mlsanity.com/paper/2311.05943/) (2023)
- [How Beginning Programmers and Code LLMs (Mis)read Each Other](https://mlsanity.com/paper/2401.15232/) (2024)
- [LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models](https://mlsanity.com/paper/2402.10524/) (2024)
- [SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines](https://mlsanity.com/paper/2401.03038/) (2024)
- [Multi-modal Synthesis of Regular Expressions](https://mlsanity.com/paper/1908.03316/) (2019)
- [Exploring Prompt Engineering Practices in the Enterprise](https://mlsanity.com/paper/2403.08950/) (2024)
- [Selenite: Scaffolding Online Sensemaking with Comprehensive Overviews Elicited from Large Language Models](https://mlsanity.com/paper/2310.02161/) (2023)
- [A Contextual Inquiry of People with Vision Impairments in Cooking](https://mlsanity.com/paper/2402.15108/) (2024)
- [ConstitutionalExperts: Training a Mixture of Principle-based Prompts](https://mlsanity.com/paper/2403.04894/) (2024)
- [Crystalline: Lowering the Cost for Developers to Collect and Organize Information for Decision Making](https://mlsanity.com/paper/2202.02175/) (2022)
- [The Evolution of LLM Adoption in Industry Data Curation Practices](https://mlsanity.com/paper/2412.16089/) (2024)
- [To Reuse or Not To Reuse? A Framework and System for Evaluating Summarized Knowledge](https://mlsanity.com/paper/2102.06231/) (2021)

## Built on by (most cited first, 1 of 1)

- [Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations](https://mlsanity.com/paper/2509.25492/) (2025)

## Abstract (arXiv)

Multimodal large language models (MLLMs), with their expansive world knowledge and reasoning capabilities, present a unique opportunity for end-users to create personalized AI sensors capable of reasoning about complex situations. A user could describe a desired sensing task in natural language (e.g., "alert if my toddler is getting into mischief"), with the MLLM analyzing the camera feed and responding within seconds. In a formative study, we found that users saw substantial value in defining their own sensors, yet struggled to articulate their unique personal requirements and debug the sensors through prompting alone. To address these challenges, we developed Gensors, a system that empowers users to define customized sensors supported by the reasoning capabilities of MLLMs. Gensors 1) assists users in eliciting requirements through both automatically-generated and manually created sensor criteria, 2) facilitates debugging by allowing users to isolate and test individual criteria in parallel, 3) suggests additional criteria based on user-provided images, and 4) proposes test cases to help users "stress test" sensors on potentially unforeseen scenarios. In a user study, participants reported significantly greater sense of control, understanding, and ease of communication when defining sensors using Gensors. Beyond addressing model limitations, Gensors supported users in debugging, eliciting requirements, and expressing unique personal requirements to the sensor through criteria-based reasoning; it also helped uncover users' "blind spots" by exposing overlooked criteria and revealing unanticipated failure modes. Finally, we discuss how unique characteristics of MLLMs--such as hallucinations and inconsistent responses--can impact the sensor-creation process. These findings contribute to the design of future intelligent sensing systems that are intuitive and customizable by everyday users.