Codecs for TouchDesigner, Explained

Working with video files is a very common (and sometimes frustrating) part of building projects in TouchDesigner. Although there are many settings available to tweak for the playback or rendering of a video within TouchDesigner, one of the most important considerations is the video codec that is used for the file. The codec you choose can make an impact on a variety of aspects of the video, including playback performance, image quality, or file size. In this post, we’ll start by going over the basics of video codecs, including what they are and the differences with a container format, and then take a look at what codecs you can use in TouchDesigner and why you might choose a particular codec over another.

Note: as this intended to a beginner’s guide, we’re not going to get super technical in this post. This guide is aimed to help new users understand the core concepts behind codecs and how they can make educated choices about which ones they might want to use.

What is a Codec?

A codec is a device or computer program that encodes or decodes a data stream or signal.¹ In our case, we’re going to be focusing on video files, so we’re going to focus exclusively on software video codecs, which compress and decompress digital video. Note that audio codecs perform a similar function for compressing and decompressing digital audio, and are often used in conjunction with video codecs (this is how video files can have sound).

As codecs are designed for different purposes, they may prioritize data compression to reduce the storage space or transmission bandwidth required.¹ As we’ll look at later, codecs are often classified into lossy and lossless codecs, which refer to the amount of information lost in comparison to the original signal.

You’re probably already familiar with more codecs that you realize. Ever heard of H.264, HEVC, FLAC or AAC? These are all commonly used codecs.

What’s the Difference Between a Container Format and a Codec?

An important distinction for beginners to make is the difference between a codec and a container. A container format is a file format that allows multiple data streams to be embedded into a single file, usually along with metadata for identifying and further detailing those streams.²

In the context of video, a container wraps the encoded video, audio, and other data like subtitles or metadata together into a single file. The image above helps to visualize this relationship. Containers are what make it easy to share and play back video files.

Containers and codecs are sometimes talked about as if they’re interchangeable (which makes things extra confusing for beginners), but in reality a container could contain a variety of different codecs. You’re probably familiar with popular container formats as well: .MOV and .AVI are two seen a lot.

For an example, take a look at the two video files above. They’re both five second long recordings of the classic jelly beans animation that displays when you open TouchDesigner. The resolution, length, and video content are exactly the same. They share the same container format (.MOV). But you’ll notice that one file is much larger in size than the other. That’s because codec_test.mov was recorded with the Animation codec and codec_test_01.mov was recorded using the H.264 codec. Even though the container format is the same, the files contain video that was encoded with different codecs. And, the chosen codec has a big impact on file size.

Video Codecs in TouchDesigner

Let’s tie this back to TouchDesigner: if you take a look at the Movie File Out TOP under the Video Codec parameter, you’ll notice there are a lot of options, including: Animation, Photo/Motion JPEG, MPEG 4 (Part 2), H.264 (NVIDIA GPU), GoPro-Cineform, Hap, Hap Q, Hap R, H.265/HEVC … it goes on!

As we mentioned earlier, codecs are created with different goals in mind: some provide a much higher quality image for long-term preservation of content at the expense of a much higher file size. Others utilize high levels of compression to greatly reduce the file size (making the file more portable and easily shared) at the expense of the image quality.

In general, you’ll see two terms thrown around when talking about the type of compression a codec uses: lossy and lossless. Lossy compression removes information from the video file to reduce file size. As information is being removed, this will result in lower image quality (although the amount is usually controllable to some extent). Lossless compression, on the other hand, does not remove information, instead using compression just to make the storage more efficient. So while the image output will be exactly what was fed into the file, the file size will generally be larger as it contains more information.

In the next section, we’ll take a look at each of the different codec options available in TouchDesigner.

Which Video Codecs are Supported in TouchDesigner?

Although the Movie File In TOP supports a few additional filetypes, we’re going to focus on the codecs that TouchDesigner can output via the Movie File Out TOP. These are some of the most common formats that you’ll use when working with TouchDesigner. Here’s a list of video codecs from Derivative’s wiki page on the Movie File Out TOP:

• Animation: Run-length encoded video, lossless codec and low decode times, but very large file size. Alpha channel can be included when Pixel Format is RGBA.
• Photo/Motion JPEG: JPEG encoded video, lossy codec and low decode times with medium file sizes. Good for playback both forwards and backwards or for random access.
• MPEG 4 (Part 2): High quality but can produce large files size and/or have high decode times.
• H.264 (NVIDIA GPU): H.264 GPU encoding, only available when using Nvidia graphics cards. Great compression and quality and small file sizes. However can suffer from high decode times and not the best for random access, scrubbing, or reverse playback. See the H264 parameter page for additional H264 encoding options.
• GoPro-Cineform: A lossy compression similar to Apple ProRes. Alpha channel can be included.
• Hap: A fast codec that uses the GPU. Lower quality than Hap Q.
• Hap Q: A fast codec that uses the GPU. Higher quality than regular Hap.
• H.265/HEVC (NVIDIA GPU): H.265 GPU encoding, only available when using Nvidia graphics cards. Better compression (and sometimes quality) than H.264 but more resource intensive to encode/decode.
• GIF: Outputs an animated, looping GIF. Image quality limitations, no audio.
• NotchLC: A high quality, GPU accelerated video format. It offers higher quality results than HapQ, at the cost of higher GPU usage as well as larger file sizes.

If you’re running a recent version of TouchDesigner (build 2022.24200 or newer), you may have noticed that there are some additional options from those listed above.

• HAP R: A 8-bit color depth movie format that has higher image quality than both Hap and Hap Q . It can include alpha.
• HAP HDR: A 16-bit color depth movie format that is just as fast to playback as other Hap formats. However, does take longer to encode.
• VP8: Codec open sourced by Google (who use it as a GIF replacement for short looped animations), predecessor to VP9.³ For encoding into .webm containers.
• VP9: Open source codec developed by Google (notably used on YouTube), competitor to HEVC/H.265.⁴ For encoding into .webm containers.

Which Codec Should I Choose?

The above descriptions give you a pretty good overview of what to expect in terms of performance, video quality and file size for each codec. But now you’re probably wondering, which one should I choose?

For starters, it’s important to know that there’s no best codec. Codec choice is highly dependent on the particular project you’re working on, and different projects will have different requirements. Some important things to consider include:

• Are you playing back or recording video content? (Prioritizing fast playback or archival recording of documentation)
• What’s the read speed of the hard drives on the production machine?
• What resolution does the content need to be?
• How will the video be played? (Forwards, Forwards/Backwards, Lots of scrubbing, etc.)

With those questions in mind, we’ll take a look a some of situations where particular codecs might be useful. Note that these are just suggestions rather than hard and fast rules, and your mileage may vary.

Playing content forwards and backwards, or doing a lot of scrubbing: Photo JPEG is great for this. NotchLC has also been touted for its scrubbing ability, but requires more storage space, a fast drive and a high powered GPU.
Also note that not all codecs are optimized for decoding forwards/backwards, so it’s important to know if this will be an aspect of your project prior to choosing the codec.

Resolution above HD: Codecs that utilize the GPU are great for larger resolutions (ex. 4k), including HAP, NotchLC, and H.265.

Resolution of HD or below: H.264 can be a good option, as long as you don’t need lossless video quality. Selecting a higher bit rate can help improve the video quality.

Limited Storage Space: H.264 or HEVC/H.265 can produce very small file size if storage is limited, at the cost of lower image quality.

Fast Drive (SSD/M.2), Content Played Forward: HAP or NotchLC offer great visual quality and performance (with a fast drive) at the cost of higher file size.

High Video Quality for Playback: HAP Q, HAP R and NotchLC all have very high quality image output and performance for real-time playback, with the caveat of high file size

High Video Quality for Archiving/Editing: The Animation codec is a go to, but H.264 and H.265 also have lossless modes available.

Again, these are just suggestions for what formats could be useful to try in particular situations. In general, it’s highly recommended to experiment with various codecs and settings to find the right fit for your project.

Wrap Up

Well, there’s no shortage of things to say about codecs! Without them, everything from binge watching shows on Netflix to recording video documentation of a project would be a lot more difficult (and require a LOT more space!). And as we’ve learned, they can definitely make an impact on your TouchDesigner projects. Hopefully this article has given you a better understanding of video codecs, what they do, how they differ, and how you can utilize them to your advantage on your next TouchDesigner project!

References

1. https://en.wikipedia.org/wiki/Codec
2. https://en.wikipedia.org/wiki/Container_format
3. https://en.wikipedia.org/wiki/VP8
4. https://en.wikipedia.org/wiki/VP9