Incorporating AI into your TouchDesigner Projects

Artificial intelligence is a powerful tool to boost our creativity. So let’s have a look at how to effectively integrate AI in TouchDesigner with some real examples.

Forget ChatGPT or the Studio Ghibli-style image generation. Artificial intelligence is about much more than solving math homework or creating memes for social media. It is a technological revolution and a game changer. Most importantly, it opens new perspectives in the visual and interactive domains. In this article we will explore how to incorporate AI in TouchDesigner. Let’s go!

AI in TouchDesigner: StreamDiffusion

StreamDiffusion is an open source tool for visual assets generation. It comes with three features: image-to-image, text-to-image and video-to-video generation. The StreamDiffusion added value is the creation of generative models in real time, thus opening interesting opportunities for its integration inside TouchDesigner.

On a technical side, using StreamDiffusion can be intimidating at first. Luckily, we can take advantage of the StreamDiffusionTD tox custom component developed by DotSimulate, that is available on download on his Patreon channel for few bucks.

StreamDiffusion runs on computers with Windows 10 or 11, NVIDIA graphic card (with CUDA support), CUDA Toolkit (11.8 or 11.2) and Git.

Stable Diffusion

Stable Diffusion is one of the most known deep learning models and can perform several tasks including – but not limited to – text-to-image, image-to-image and image-to-video generation. It is available online but it can be accessed via APIs as well through the Developer Platform website. API usage is based on credits and price varies according to the service.

Stable Diffusion is a powerful and complex ecosystem that runs on three main models. The Structure model creates images while keeping the original structure of the original image. The Style model extracts elements from the input image and generates a new image based on the prompt content. The Stable Fast 3D model creates 3D assets form two input images.

We can get the most out of Stable Diffusion in TouchDesigner by integrating it into our patch pipeline via APIs. You can find two dedicated tutorials here and here.

LoRA: your own personal AI

One of the most fascinating aspects of AI is that every day new features come alive. Think about AI models, for example. Now we can create our own models to train our favourite AI systems and get the outcome in TouchDesigner.  

This magic is called LoRA – Low Rank Adapatation – a technique that allows us to adapt machine learning models into new models. LoRA adds new lightweight information on existing large models, without the need to change the entire model.

There are several tools we can use to create our own LoRA. I suggest you read these two tutorials here and here.

But let’s get a real example. I created my own LoRA by training a model with high resolution scans of a friend paintings. Then I used it in StreamDiffusion in TouchDesigner and I got a real time unique generative installation.

Beyond generative AI

Generative AI is surely one of the most fascinating topics of artificial intelligence. But there are hundreds of cool machine learning applications out there. One example is computer vision.

To get an example, we can use computer vision algorithms to perform people detection and track their position. This can be done with YOLO – You Only Look Once – a clean and fast real time detection algorithm that is able to detect objects and people with a single look.

YOLO algorithms work well in Python and need just a simple camera. Here below is a sample code. For sake of simplicity, I ran the script outside TouchDesigner, but we can easily use it with a Text or Script DAT.

from ultralytics import YOLO
import cv2
from pythonosc.udp_client import SimpleUDPClient
import time

osc_client = SimpleUDPClient("127.0.0.1", 7000)

model = YOLO("yolov8n.pt") #Remember to download the model

cap = cv2.VideoCapture(2)

while True:
    ret, frame = cap.read()
    if not ret:
        break

    results = model(frame, verbose=False)[0]

    people = [d for d in results.boxes.data.cpu().numpy() if int(d[5]) == 0]
    count = len(people)

    osc_client.send_message("/person/count", count)

    for i, person in enumerate(people):
        x1, y1, x2, y2, conf, cls = person
        cx = (x1 + x2) / 2
        cy = (y1 + y2) / 2

        osc_client.send_message(f"/person/{i}/pos", [float(cx), float(cy)])

        cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)), (0, 255, 0), 2)
        cv2.putText(frame, f'Person {i}', (int(x1), int(y1 - 10)), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0,255,0), 1)

    cv2.imshow("YOLOv8 Detection", frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Wrap Up

There is no doubt that artificial intelligence has revolutionized content generation. It is an ever changing landscape and new tools are available every day. While some technical aspect can be intimidating at first, using AI in TouchDesigner opens new promising perspectives. As usual, the sky is the limit. So feel free to experiment, experiment and experiment!