# CLEAR: Cue Learning using Evolution for Accurate Recognition Applied to Sustainability Data Extraction

arXiv: https://arxiv.org/abs/2501.18504 · HTML: https://mlsanity.com/paper/2501.18504/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 30 Jan 2025.

## Builds on (most cited first, 17 of 17)

- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [Large Language Models as Evolutionary Optimizers](https://mlsanity.com/paper/2310.19046/) (2023)
- [Language Model Crossover: Variation through Few-Shot Prompting](https://mlsanity.com/paper/2302.12170/) (2023)
- [LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization](https://mlsanity.com/paper/2306.01102/) (2023)
- [Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes](https://mlsanity.com/paper/1706.03673/) (2017)
- [MarioGPT: Open-Ended Text2Level Generation through Large Language Models](https://mlsanity.com/paper/2302.05981/) (2023)
- [Investigating Prompt Engineering in Diffusion Models](https://mlsanity.com/paper/2211.15462/) (2022)
- [A Taxonomy of Prompt Modifiers for Text-To-Image Generation](https://mlsanity.com/paper/2204.13988/) (2022)
- [Best Prompts for Text-to-Image Models and How to Find Them](https://mlsanity.com/paper/2209.11711/) (2022)
- [LLM Guided Evolution -- The Automation of Models Advancing Models](https://mlsanity.com/paper/2403.11446/) (2024)
- [EIVEN: Efficient Implicit Attribute Value Extraction using Multimodal LLM](https://mlsanity.com/paper/2404.08886/) (2024)
- [SCAPE: Searching Conceptual Architecture Prompts using Evolution](https://mlsanity.com/paper/2402.00089/) (2024)
- [Address-Specific Sustainable Accommodation Choice Through Real-World Data Integration](https://mlsanity.com/paper/2405.12934/) (2024)
- [Automated Real-World Sustainability Data Generation from Images of Buildings](https://mlsanity.com/paper/2405.18064/) (2024)

## Abstract (arXiv)

Large Language Model (LLM) image recognition is a powerful tool for extracting data from images, but accuracy depends on providing sufficient cues in the prompt - requiring a domain expert for specialized tasks. We introduce Cue Learning using Evolution for Accurate Recognition (CLEAR), which uses a combination of LLMs and evolutionary computation to generate and optimize cues such that recognition of specialized features in images is improved. It achieves this by auto-generating a novel domain-specific representation and then using it to optimize suitable textual cues with a genetic algorithm. We apply CLEAR to the real-world task of identifying sustainability data from interior and exterior images of buildings. We investigate the effects of using a variable-length representation compared to fixed-length and show how LLM consistency can be improved by refactoring from categorical to real-valued estimates. We show that CLEAR enables higher accuracy compared to expert human recognition and human-authored prompts in every task with error rates improved by up to two orders of magnitude and an ablation study evincing solution concision.