So to kick off this blog right I have to quickly mention one thing. Why would someone want to use both these applications at once?
The short answer is because they both have their strengths and being able to put them together means you make pretty much any piece of hardware, data signal, or sensor able to render with the same graphics that AAA game titles use.
Have a web server that you want to ping for live updates to a real time weather system? What about Media Pipe running pose detection on a webcam affecting a whole smattering of 3D physics objects? And how about running an Arduino controller feeding live positional and rotational data to a 3D object so that there’s a digital twin registering every flick of the wrist or wave of the arm?
Well you can do all this and more.
Sure, some out there might be thinking that you could ditch Unreal Engine in favor of the in built rendering tools in TouchDesigner. And they’d be right. But those tools have to be pushed in so many incremental ways to get good, professional level rendering setups.
Treating TouchDesigner as the way station for connecting whatever you want under the sun and feeding that into Unreal allows for so many creative possibilities that it’s hard to know where to even start.
In this post, we’ll cover the different ways you can connect TouchDesigner and Unreal Engine, why you would want to do so, and some examples of projects you can get started on today.
Protocols
Mastering the communication between these two apps is the key. So when thinking about data being passed back (and forth), you have a few tried and true options to choose.
OSC and UDP
So if you want to send anything like real time values for a sensor, state driven variables driven by Touch, or anything text/string related, then realtime protocols like OSC or UDP are your best bet.
I find myself always reaching for OSC in my workflows since the realtime packets of data support all your favorite variable data types. The only big headache is managing IP addresses and ports, especially when there’s multiple machines in an installation. But it’s the most utilized.
Both these protocols are available for free for you to use in your next project. So there’s no paywall keeping you from piping state driven data from Touch straight into a sick looking Unreal render.
NDI/Syphon Spout and TouchIn/Out
Now whenever you’re wanting to send over a live video feed, texture, video playback, or textured driven particle system, then NDI or Syphon are the protocols you want to reach for.
NDI is great and highly supported whether for on or off machine communication but you can be limited in the precision of values (8, 16, or full 32bit float values) in the actual textures and the latency can be noticeable for some applications.
When sending over a video for playback on, say, a texture in Unreal Engine, this will be hardly noticeable. And whether the texture has 8bit vs 32bit precision (this means that there more values for each pixel in the texture to be) will mostly not matter. But if you’re going to be sending over a data texture where each pixel tracks the position or state of a particle system, then 32bit precision and as low of latency as possible is the holy grail.
This is where Syphon/Spout comes in. This protocol is only limited to communication between applications on the same machine. So if you have a single computer running both Touch and Unreal, then you’re good to go. This will usually support the full 32bit float precision we like, and the textures are stored and retrieved directly from the GPU, so it’s blazingly fast.
Now there’s a middle ground for these and is important to note. If you have a setup where Unreal is on one computer and your main Touch machine is on another, you have the option to TouchOut from the main machine to the other (preferably with an ethernet cable for the lowest latency) and receiving that onto the machine using Unreal Engine. HOWEVER, that second machine must retrieve that TouchOut texture with another instance of Touch. Then you would need to convert that into a Syphon/Spout texture for Unreal to consume.
This setup isn’t as simple as NDI but the two might be comparable in speed and dependent on your setup. But I thought it’d be good to mention.
Get Our 7 Core TouchDesigner Templates, FREE
We’re making our 7 core project file templates available – for free.
These templates shed light into the most useful and sometimes obtuse features of TouchDesigner.
They’re designed to be immediately applicable for the complete TouchDesigner beginner, while also providing inspiration for the advanced user.
And Finally for the Fun Stuff
So let’s zoom in now on some applications for Touch and Unreal to create something and get those mental gears turning for your next project.
Computer Vision
One of the strengths of TouchDesigner is how easily we can plug in different Computer Vision toolsets. Whether it’s using OpenCV, YOLO, Kinect, or MediaPipe, there’s a general ease with which we can deploy pose or object tracking into our networks.
Well what if we were to take MediaPipe and apply it to an external camera and extracted out the positions of our hands, arms etc. and then streamed those positions over via OSC into Unreal Engine.
Once we have those positions, we could connect them to realtime objects interacting with a fluid simulation driven by Fluid Ninja. Instead of just flapping your arms at some good looking particles, we can have a really cool looking fluid simulation by combining these two tools together.
As an additional aside, there’s an awesome installation to be made that combines a fluid sim with an interactive LiDar driven surface that uses the same ideas as this Media Pipe workflow. However, you would most likely be sending over a texture map of the LiDar surface.
Hardware
Dipping into the world of hardware reaps tons of possibilities for Touch-Unreal integration. Live sensors could track an object’s position or rotation and that in turn could be synced to a light position in an Unreal project so that users are live interacting with the digital environment.
What if you had a room with projectors displaying an Unreal Engine scene and physical objects in the room affected digital objects in the scene. Conceivably, you’d just have small arduinos connected to a light switch that broadcasts a signal to Touch that in turn broadcasts to Unreal Engine to turn on the light. Or a flashlight enables people to illuminate the scene displayed on the room around them. This would work the same way where Touch acts as the brains of gathering the hardware data, and passing that over into Unreal Engine.
Web API
Finally, Touch supports querying APIs so what if we took a public API like Windy that tells us live data about wind conditions in a specific location and fed that into Unreal Engine using Touch as a way station. We’d use OSC again, but say it’s driving a windy scene we’ve built in Unreal Engine.
I can absolutely imagine a project proposal where an LED screen in a building displays our live scene gathering weather data and mirroring the realtime weather conditions that a passersby sees outside the window.
Our installations come closer to being a clone of the world we live in and breathe in.
Wrap Up
Hopefully this post has given you a good understanding of the different ways that we would want to connect TouchDesigner and Unreal Engine and why!
For even more TouchDesigner and Unreal Engine integrations, watch my two-part tutorial series here:
