Researchers of the Innovation Cube at the School of Communication, Media and Information at Fundação Getulio Vargas (FGV ECMI) are conducting experiments with Stable Diffusion, a generative artificial intelligence model that produces images using ComfyUI. The experiments consist on capturing real-time images and then processing them using ComfyUI, which creates a new image using techniques such as rotoscoping. This new image is based not only on the original captures, but also on search instructions provided through a textual prompt and filters.

The final goal is to conceive a multisensory piece to compose the Digital Art Biennial exhibition, which will happen in 2024 and 2025 at Futuros – Arte e Tecnologia in Rio de Janeiro with the theme DEEP SLEEP. The project is a partnership between the Innovation Cube and another FGC ECMI project involving the Media Lab, the school’s laboratory focusing on audiovisual content.

During the experiments, the researchers found that even when using a very detailed set of words to exclude from the production (a negative prompt), many inappropriate images were generated, such as naked bodies or firearms. For that reason, after extensive testing in order to adapt the content to the institution’s policies and the age range of the target audience, and taking into account the theme and goals related to dreams, a model based on crayon illustrations is being used.

The Stable Diffusion model is free and can be used as a desktop application or in any browser. The browser-based version enables third parties to perform the coding and is an option for computers with lower computational power.  

Since rendering relies heavily on computational processing speed, the process has a delay of approximately two seconds in order to produce a result that can be viewed in an image with a resolution of 1024x576px when using the Innovation Cube computers.

For those not familiar, ComfyUI is a Graphical User Interface (GUI) based on points or nodes that generates image results through fine-tuning.