What is Stable Diffusion and How to Use it in TouchDesigner

In this tutorial we will learn how to integrate Stable Diffusion in TouchDesigner to create AI-powered creative projects. Let’s go!

Generative artificial intelligence is paving the way for an explosive revolution in all the fields of human activity. From chatbots to predictive medicine, AI is unleashing promising opportunities. Not to mention in the interactive and immersive domain.

A great example of generative AI is Stable Diffusion. It is a deep learning model that can be used to generate images and videos from simple prompts. Stable Diffusion is available as an online tool and it can be integrated into wider projects via APIs.   

Stable Diffusion APIs are available upon registration on the Developer Platform website. Here we can create API keys and buy credits. API usage is based on credits, free registration includes 25 free credits and one credit corresponds to $0.01. Price varies according to the service.

So, in this tutorial we will integrate Stable Diffusion in TouchDesigner in two ways. First of all, we will create a text-to-image generation system. Then, we will use that to generate a video for further manipulation. So, let’s start patching!

Image generation with Stable Diffusion in TouchDesigner

So, first of all we will generate an image with Stable Diffusion inside TouchDesigner. We will use Stable Image Ultra, which is the most advanced text to image generation tool available.

In order to do so, we need to create a CHOP Execute DAT that runs the Python script each time a button is pressed.

Here is the code:

---

import requests

def onOffToOn(channel, sampleIndex, val, prev):
	return

def whileOn(channel, sampleIndex, val, prev):
	return

def onOnToOff(channel, sampleIndex, val, prev):
	return

def whileOff(channel, sampleIndex, val, prev):
	return

def onValueChange(channel, sampleIndex, val, prev):

	response = requests.post(
	    f"https://api.stability.ai/v2beta/stable-image/generate/ultra",
	    headers={
	        "authorization": "INSERT-YOUR-API-KEY",
	        "accept": "image/*"
	    },
	    files={"none": ''},
	    data={
	        "prompt": op('prompt').par.text,
	        "negative_prompt": op('negative_prompt').par.text,
	        "output_format": "png",
	        "aspect_ratio": "1:1",
	    },
	)
	
	if response.status_code == 200:
	    with open("./image.png", 'wb') as file:
	        file.write(response.content)
	else:
	    raise Exception(str(response.json()))
	
	
	return

---

Let’s analyze it.

First of all we import the requests library to send HTTPS requests. Then, in the onValueChange function, we call the API function and our API key. In the next lines we design our positive prompt – what we would like to see in the image – and the negative prompt – what we would not like to see in the image – then define the output format and the aspect ratio. By clicking the button, the script is launched and the image is saved in our project folder.

The default aspect ratio is 1:1 (1024 x 1024), other ratios available are 16:9, 1:1, 21:9, 2:3, 3:2, 4:5, 5:4, 9:16, 9:21. The default output format is .png, other formats available are .jpeg and .webp.

Going back to the TouchDesigner patch, the flow is quite simple.

By pressing the leftmost button the script will run and the image will be generated and stored inside the project folder.

That’s all! But what if we would like to immediately display the generated image in TouchDesigner? And what if we would like to generate a video from the image?

In order to immediately display the image, we can:

• Beforehand, place a placeholder image inside the folder with the same name of the generated image: in my case, image.png
• Create a Movie File In component that points to the placeholder image in the file path and connect the reload/pulse selector to a button

Once the image has been generated, just click the button and the image will be displayed.

Image to video generation with Stable Diffusion in TouchDesigner

Stable Diffusion offers an image-to-video generation tool as well. So we can easily integrate it in our project pipeline in TouchDesigner.

It is important to underline that only three supported dimensions are currently available: 1024×576, 576×1024, 768×768. Trying to generate videos from unsupported image dimensions will turn into errors.

Therefore, if we want to generate a video from the image we just generated inside TouchDesigner, we have to add some steps.

We need to connect a Null component to the Movie File In output and connect it to a Resolution component. Change the resolution to 768×768, connect the output to a Null component and then to a Movie File Out component. Here, in the file Path where we point to the name of our image, select Image in the type box and click the Record button. The image will be saved with the right format for video generation.

The image to video process is divided into two sub-processes: the Start generation and the Fetch generation result.

In the first one the video is generated, in the second one the video is retrieved and can be added to TouchDesigner for further manipulation.

This is the Start generation Python code:

---

def onOffToOn(channel, sampleIndex, val, prev):
	return

def whileOn(channel, sampleIndex, val, prev):
	return

def onOnToOff(channel, sampleIndex, val, prev):
	return

def whileOff(channel, sampleIndex, val, prev):
	return

def onValueChange(channel, sampleIndex, val, prev):
	import requests

	response = requests.post(
	    f"https://api.stability.ai/v2beta/image-to-video",
	    headers={
	        "authorization": f"INSERT-YOUR-API-KEY"
	    },
	    files={
	        "image": open(r"INSERT-YOUR-FILE-PATH", "rb")
	    },
	    data={
	        "seed": 0,
	        "cfg_scale": 1.8,
	        "motion_bucket_id": 127
	    },
	)
	
	print("Generation ID:", response.json().get('id'))
	op('video_id').par.text = response.json().get('id')

	return

---

The code outputs a 64-characters string that is displayed in a Text component.  

This is the Fetch generation result Python code:

def onOffToOn(channel, sampleIndex, val, prev):
	return

def whileOn(channel, sampleIndex, val, prev):
	return

def onOnToOff(channel, sampleIndex, val, prev):
	return

def whileOff(channel, sampleIndex, val, prev):
	return

def onValueChange(channel, sampleIndex, val, prev):
	import requests
	
	generation_id = op('video_id').par.text
	print(generation_id)
	
	response = requests.request(
	    "GET",
	    f"https://api.stability.ai/v2beta/image-to-video/result/{generation_id}",
	    headers={
	        'accept': "video/*",  # Use 'application/json' to receive base64 encoded JSON
	        'authorization': f"INSERT-YOUR-API-KEY"
	    },
	)
	
	if response.status_code == 202:
	    print("Generation in-progress, try again in 10 seconds.")
	elif response.status_code == 200:
	    print("Generation complete!")
	    with open("video.mp4", 'wb') as file:
	        file.write(response.content)
	else:
	    raise Exception(str(response.json()))
	return

---

Here, the 64-character string is automatically added to the API URL in order to automate the whole process. The video is then automatically downloaded in our project folder.

In order to display the video inside TouchDesigner, we can almost proceed in the same way we did for the image generation part, so:

• Beforehand, place a placeholder video inside the folder with the same name of the generated video: in my case, video.mp4
• Create a Movie File In component that points to the placeholder video in the file path and connect the reload/pulse selector to a button

So, if we run the two scripts sequentially, in a short span of seconds we will have a brand new video at our disposal. Of course we can manipulate the video inside TouchDesigner according to our needs and goals.

It is important to underline that the currently supported dimensions are quite limited, but Stable Diffusion offers upscaling features as well to upscale images up to 40x while preserving all aspects.

Wrap Up

Stable Diffusion is a powerful tool to integrate generative artificial intelligence into our creative pipeline with TouchDesigner. Possibilities are endless and the whole technical workflow is quite simple to implement. As usual, sky is the limit!

For a more advanced look at using Stable Diffusion with TouchDesigner, read our next post here.

Download the patch here