If you told me a few years ago that you could get a real-time background subtraction with no special camera hardware and it can work just about anywhere, I’d be pretty skeptical. While not perfect by any stretch, Nvidia’s new machine-learning driven background removal is pretty fantastic. In TouchDesigner, we have native access to the Nvidia Background TOP, but sometimes you might want to get access to these features outside of TouchDesigner or do this processing outside of TouchDesigner, and that’s where Nvidia’s Broadcast App comes into play.
Why’s it useful?
Nvidia did a really smart thing here and looked at how the community uses these tools (something I wish Microsoft would take note of when it comes to Kinect development). Not only did they release the SDK so other applications can add the features natively, they also took the same SDK and wrapped it around a simple app that takes your camera/mic inputs, process them, then creates a virtual webcam with the results. This means you can access these features quickly and easily in basically any application or environment, whether you’re working in TouchDesigner or just hopping on a Zoom call.
The only requirement to using these new Nvidia features is to have a relatively recent Nvidia RTX GPU. This doesn’t mean you need the latest and greatest, thankfully. Nvidia states that you’ll need:
NVIDIA GeForce RTX 2060, Quadro RTX 3000, TITAN RTX or higher
The RTX 2000 series came out at the beginning 2019, roughly just over 2 years ago, so if you’ve upgraded your system since then or are creating a system for a project, it’s likely you’re going to be using something from the RTX 2000 series or newer. Other than that, no requirements, which is great.
The first thing you’ll need to do is install the Broadcast App. You can download it below:
You’ll also want to make sure you’re on the latest Nvidia driver which you can get at the link below:
Setting up your inputs
While we’re mainly interested in the background removal currently, there are some great audio filters in the Broadcast app. The first screen you’re greeted with on launch is the Microphone page:
Before we dive into the actual effects, the first thing you’ll want to do is select your microphone in the top left source area. Then let’s switch over to the Camera tab at the top and select our proper source device:
Once you’ve selected the camera you can set the desired resolution and FPS, and the button circled in red allows you to edit your camera’s settings like focus and brightness, etc. Once you’re done that, we’ve got our inputs setup!
The nice thing about the Broadcast App is that it uses a pretty uniform interface. Once we learn how the effects work on the camera section, you can easily switch over to microphone page and repeat the process with those effects. Today we’ll focus on the juicy video effects.
Under the word effects, you can think of this area like a stack of plugins. You can add multiple effects and put them in any order you like. If you’d like to start with a clean slate, or if you want to delete any effects, you can click on the name of the effect and you’ll see x Remove this effect:
Clicking on that will remove that effect from the stack and you’ll be starting fresh:
So let’s add our first effect by clicking Add Effect, selecting Background Removal, and then we’ll turn on the toggle button next to it:
Boom! That’s it! You can try different effects by switching them in the drop down menu:
And like we mentioned you can stack effects. Let’s say we want to use the background removal AND have it auto-frame the feed to follow our face. We can hit Add Effect again and drop in a second block:
All you have to make sure to do is enable the toggle next to the effect. Great! We’ve got a key and an auto-frame setup in a few minutes. Now what?
Get Our 7 Core TouchDesigner Templates, FREE
We’re making our 7 core project file templates available – for free.
These templates shed light into the most useful and sometimes obtuse features of TouchDesigner.
They’re designed to be immediately applicable for the complete TouchDesigner beginner, while also providing inspiration for the advanced user.
Bringing the effects into your app
Once we’re done our work in the Broadcast App, we can dive into your favourite app. In this case I’ll open TouchDesigner and drop a Video Device In TOP. When I look at the available devices, we can see a new one called Camera (NVIDIA Broadcast):
Once you do that we’ll see that same feed, also we can notice that it’s not just a black background, but it actually has an alpha channel!
If you’re using a computer that has an RTX graphics cards, you can see how quickly and easy it is to get background removal into any app you’re using and working in. While these features are still new and have some rough edges, they’re extremely promising! The fact that they’re so easy to use already makes them something we’ll be using in our projects and that you should be keeping your eyes on!