Designing for Visual Thinkers: Overcoming Text-Centric Limitations in GenAI Tools

Abstract. This workshop paper examines the challenges of integrating generative AI (GenAI) into the design process, focusing on designers who are predominantly visual thinkers. Drawing from previous interviews with designers, we highlight challenges related to text-based inputs, difficulties in controlling prompts, and the lack of proper integration into design practices. Our paper explores ways to improve designers’ interaction with GenAI tools, enabling them to achieve desired outcomes with less effort. We propose two approaches: one that enhances the use of text-based inputs through improved prompt generation with multimodal Large Language Models (LLMs) and another that introduces more intuitive, visually-driven methods like sketch and doodle inputs, reducing reliance on text prompts. This paper aims to foster discussions on how GenAI tools can better align with the needs of design professionals. Our work thus focuses on addressing the limitations of text-centric GenAI interfaces, making these tools more accessible and effective for visual thinkers, and ultimately improving their ability to leverage GenAI in the creative process.