Intuitively communicating with generative AI through Promptac

Bridging the gap between the digital and physical world – Royal College of Art graduate Zhaodi Feng presents an interactive toolkit Promptac that allows users to input prompts to AI image generators utilizing hand movements rather than text. Intrigued?

A multidisciplinary designer – she aims to design simple and light-hearted entry points to the complexity and contradiction of global issues, communicating through interactions and engaging people in reflecting the relationship between the tradition and the future.

Today, technology advancements have revolutionized the way we create, shifting from traditional crafts to digital design and now to AI-assisted generative design. However, as we spend more time in the digital realm, we may question if these innovations have disconnected us from the physical world. 

From a fingertip controller that recognises materials and colors from the real world to small tools that users can twist, pinch, press and bend to shape changes in AI-generated digital designs – the tools in the exploratory Promptac kit innovatively act as a translator between the user’s actions and AI crafted from foam clay and silicone.

Today, most AI image generation tools require precisely written text describing materials, textures, shapes and colors to generate the desired image. Keeping non-designers in mind, she hopes the tools make it easier to communicate design changes without the precise language needed.

“This kit is designed to reduce the reliance on explicit texts in modifying prompts, especially for people who rely more on sensory experiences and hand-object interaction to perceive and express their ideas.

There is a potential for improving co-design communication efficiency with this kit because it can enable non-designers and clients to naturally communicate desired materials, shapes  and colors to designers without a specific vocabulary of design terms.

I think Promtac further has the potential to benefit people with difficulties or disabilities in text communication and may help promote inclusiveness of technology in the potential future scenario of widespread usage of generative AI tools!” shares Feng.

Additionally, she went on to test the prototype products utilizing AI tools Midjourney and InstructPix2Pix that were used to generate a basic image of an object by entering a text prompt, such as a vase, mug or plant pot.

This image can then be further modified by interacting with the Promptac toolkit where – the fingertip controller is utilized to single out materials and colors from the real world to apply to the AI-generated object.

Generating a text prompt to the AI and changing the appearance of the object in the image – the controller embodies a color sensor and a radio frequency identification (RFID) sensor that recognizes a palette of RFID-tagged materials.

Similarly with the other Promptac tools – which Feng calls soft sensors – users can apply physical gestures that are effortlessly translated into text prompts to alter the shape of a 3D model representing the AI-generated image.

“The twist sensor creates a spiral shape, the pinch sensor forms an irregular change of shape and makes it longer or narrower, the press sensor flattens the shape and the bend sensor forms a curved or angled shape!” she adds. 

The soft sensors were fabricated in order to allow users to modify AI images in ways that imitate making things by hand – just like playing with clay. She began developing Promptac while researching traditional crafts and was informed by the evolution of design tools and the ways that technological advances change how people create.

“The influence of AI on design and creativity depends on how it is designed to be utilized. We need a better touchpoint to interact with AI and let technology help creativity flow!” concludes Feng.