Have you integrated NVIDIA Maxine’s AI features into TouchDesigner yet?
By this point, NVIDIA Maxine has been out for about a year, but TouchDesigner developers sometimes look over its AI features. Maxine has valuable qualities that allow you to do fancy camera tricks without needing a Kinect sensor and does so using GPU-accelerated processes.
What is NVIDIA Maxine?
NVIDIA is a suite of GPU-accelerated SDKs and cloud-native microservices that uses AI to enhance audio, video, and augmented reality effects for real-time communication services and platforms. While these are originally targeted to real-time communication, they have a huge usability in live broadcast and real-time immersive media installations because of the features they make readily accessible to computers with Nvidia RTX hardware.
NVIDIA Maxine Features
Video Effects include background removal without a green screen and face tracking. Audio Effects include background noise removal, room echo removal, and they have very impressive keyboard typing noise removal. Here are two example videos showing off some of these features:
- Windows 10, Windows 11 64-bit, Ubuntu 18.04, Ubuntu 20.04, CentOS 7
- Windows SDK: NVIDIA GeForce RTX 20XX and 30XX Series, Quadro RTX 3000, TITAN RTX, or higher (any NVIDIA GPUs with Tensor Cores)
- Server SDK: V100, T4, A10, A30, A100 (with MIG support)
- Support for Ada-generation GPUs for Windows SDKs
Using NVIDIA Maxine
Okay, you have an NVIDIA RTX card, but how do I use the feature? Download NVIDIA Broadcast; it’s free!
Once you open the App, there are three tabs on top ‘MICROPHONE’, ‘SPEAKERS,’ and ‘CAMERA’.
Select your Microphone Source, and you can turn on ‘Noise Removal’ and ‘Room echo Removal’ with the toggle switches on their right side. You can test how the audio sound with the recording and play UI on the right side. Adjust the Strength to your liking. From my experience, if there’s a lot of background noise and the noise and room echo strength are up high, there is some audio clipping if my voice is too loud, so experiment to find what works best for you in your environment.
Go to the ‘CAMERA’ Tab. Select your camera source. Switch on the Background removal toggle, and Ta-DA, the background is out!
You can also chain together the different video effects by clicking ‘ADD EFFECT’ and selecting the noise reduction or auto framing.
Using this feed
You can set the microphone and camera settings in your video calls or streaming by selecting ‘NVIDIA Broadcast’ as your audio and Camera device.
Now your Camera and Audio output are using NVIDIA Maxine features inside of any application that has webcam support!
You don’t need to use a Kinect to remove the camera background anymore! There are places where that might still be beneficial, but for general production environments, this can be a lot easier and straight forward. There are two ways to integrate these features.
Nvidia Broadcast device
This method works similarly to how you would add this effect to any app that supports a webcam input. I find this method also ends up working well because Nvidia Broadcast app handles a lot of the tricky parts of the setup for you, like ensuring your video feeds are frame synced, etc.
Drop a Video Device In TOP and change ‘Source’ to ‘Camera (NVIDIA Broadcast)’ and see your background removed or auto framed as you would inside of the Nvidia Broadcast app.
You can easily composite layers behind the camera input or add TOP effects onto the live camera feed.
You can do similar with the webcam audio as well if you use an Audio Device In CHOP and set the Device to ‘Microphone (NVIDIA Broadcast)’:
You can then use your TouchDesigner output as your virtual call or streaming output with NDI Tools. You can put an NDI Out TOP at the end of your Network and select NDI as the camera output. Then you can pickup that stream using NDI Tool’s Virtual Camera tools.
Get Our 7 Core TouchDesigner Templates, FREE
We’re making our 7 core project file templates available – for free.
These templates shed light into the most useful and sometimes obtuse features of TouchDesigner.
They’re designed to be immediately applicable for the complete TouchDesigner beginner, while also providing inspiration for the advanced user.
Native Integration in TouchDesigner
One of the great new features in TouchDesigner is the native integration of Nvidia Maxine SDK as operators. For example there are the Nvidia Background TOP and Nvidia Denoise TOP that help you do the same things that Nvidia Broadcast App does. There are also Face Track CHOP to help you work through face tracking.
If you want to dive into native integration of background removal, this video covers everything you need to know:
To dive into denoising live camera feeds in TouchDesigner, this video gets you up and running quickly:
With NVIDIA Maxine, you can also do Face tracking in TouchDesigner using the Face Track CHOP which you can dive into here to learn how to use it as well as expand on it by making AR filters with the data:
You can also learn more about TouchDesigner and NDI workflows that can help you send your processed signals and new content to different locations and applications:
The NVIDIA Maxine makes it easier to create an advance level video project in TouchDesigner without the need for advance level sensors. If you are already using NVIDIA RTX GPUs for their graphics processing power, make sure to not miss out on this advanced functionality.