Updated ZED Camera Features in TouchDesigner

Greetings, TouchDesigner developers! As the year draws to a close, we’re excited to dive into the world of ZED cameras and their integration with the latest TouchDesigner update. This post will explore the key new features of ZED cameras and how they can be implemented in our TouchDesigner workflow. Lastly, we will offer inspiring examples of how they can be creatively applied in your projects. Let’s get to it!

What are ZED cameras?

ZED cameras, developed by Stereolabs, represent a leap forward in camera technology. Unlike conventional cameras, these are equipped with dual lenses, enabling them to perceive depth and capture spatial information. This dual-lens system allows for advanced depth perception, 3D mapping, and spatial understanding. The key advantage of these cameras lies in their ability to not just capture images but to understand the environment around them. This makes them particularly useful for applications that require real-world spatial awareness, such as augmented reality, robotics, and interactive installations.

Main Features

1. Dual-Lens Stereo Vision: Provides advanced depth perception and 3D mapping capabilities.
2. Spatial Understanding: Offers a detailed understanding of the surrounding environment.
3. Motion Tracking: Tracks objects and people in real-time with high accuracy.
4. High-Resolution Imaging: Captures high-quality images, essential for detailed visual work.
5. Robust Build: Designed for a variety of environments, enhancing versatility.
6. Integrated Sensors: Includes IMU, barometer, and magnetometer for comprehensive data collection.
7. Flexible Connectivity: USB 3.1 connection for easy integration with various systems.

If you haven’t had a look at these cameras in action, here are two videos by Elburz on the camera’s introduction and 3D scanning capabilities:

ZED Cameras and TouchDesigner

Integrating ZED cameras with TouchDesigner is a seamless process! Through the ZED TOP, ZED CHOP and ZED SOP operators, TouchDesigner harnesses the cameras’ dual-lens capabilities for advanced visual and interactive projects. Whether creating immersive environments, interactive installations, or augmented reality experiences, these bridging operators offer a robust platform for complex visual creations. Let’s take a look at them individually!

ZED TOP

The ZED TOP operator in TouchDesigner allows users to capture and process various types of data from the camera, including color images, depth maps, and point clouds. This operator enables a range of options, from selecting camera views to adjusting image quality and depth settings. It’s particularly useful for projects that require real-time 3D rendering, depth-based effects, or interactive environments. It mainly provides a gateway to harnessing the full potential of ZED cameras within the TouchDesigner platform.

ZED CHOP

The ZED CHOP operator in TouchDesigner extends the functionality of ZED cameras by focusing on motion tracking and data analysis. It captures and processes positional and plane data from the camera, providing real-time tracking of objects, people, and their movements. This operator is invaluable for projects involving body tracking, gesture recognition, or spatial analytics. Its precise tracking capabilities enable the development of more interactive and responsive environments, enhancing the viewer’s experience in real-time installations and performances!

ZED SOP

The ZED SOP in TouchDesigner brings a unique dimension to working with ZED cameras. It specializes in generating 3D geometry from the camera’s depth data. This operator can create meshes and point clouds, allowing designers to incorporate real-world spatial data directly into 3D environments. It’s particularly effective in scenarios where real-time 3D visualization is essential, such as in virtual reality or augmented reality applications. By converting depth data into usable 3D geometry, this SOP broadens the creative horizons for projects that blend physical and digital elements.

Notable improvements

Full Skeletal Body Tracking (ZED CHOP): This feature provides advanced body tracking capabilities. It can track multiple people simultaneously, capturing detailed movements of the entire skeleton. This is ideal for projects that require precise human interaction, such as performance art, interactive installations, and virtual reality experiences.

With the discontinuation of Kinects, a new solution for skeleton tracking has become essential, and the ZED hardware and software step in to fill this gap. Unlike Kinects that relied on infrared (IR) technology, which can be less effective in sunlight, ZED cameras use RGB cameras for body tracking. This makes them suitable for outdoor use as they are not easily washed out by the sun.

In terms of tracking detail, the ZED CHOP supports various body formats. These include BODY_18, BODY_34, and BODY_38, each offering different numbers of keypoints for detailed movement capture. These formats can track up to 18, 34, and 38 keypoints respectively, providing comprehensive data for body movement analysis. This level of detail makes ZED CHOP an invaluable tool for applications needing precise human interaction tracking, both indoors and outdoors.

Lastly, according to the official documentation and forum sifting, the limit to the number of bodies that can be tracked is only limited by computational capacity! Meaning that with a powerful GPU tracking numerous bodies shouldn’t be a problem at all!

Neural Quality Depth Output (ZED TOP): Traditional depth sensing in ZED cameras, while innovative, has faced challenges like noise in the depth map, especially under varying lighting conditions. The integration of neural networks has significantly improved this aspect. These networks aid in producing smoother, more reliable depth maps, even in less-than-ideal lighting or environmental conditions.

This depth sensing advancement leads to more accurate and detailed depth perception, essential for intricate 3D visualizations and augmented reality experiences. The depth can be captured at longer ranges, from 0.2m up to 20m, and at a higher frame rate of up to 100 FPS, enhancing its utility in diverse scenarios.

4 General Ideas to Get Started with ZED cameras

The process of becoming a ZED camera magician is certainly broad. However, there are small projects you can start with in order to get familiar with the workflow of this tool and its integration with TouchDesigner. Remember that getting the basics right is crucial to achieve larger and more complex ideas!

• Interactive Art Installation: Use the ZED camera’s depth sensing to create visuals that change based on viewer proximity or movement. This can be as simple as controlling objects, or geometric shapes’ scale and color with your body.
• Simple Augmented Reality: Overlay digital graphics onto the real world using the camera’s depth data. Imagine taking a live scene and overlaying foreign objects and data onto it!
• Gesture-Controlled Interfaces: Utilize body tracking to create interfaces that respond to hand or body movements. With the full skeletal tracking being supported you can achieve complex and engaging visuals tailored for performance and live-art.
• Environmental Mapping: Experiment with 3D mapping to visualize spaces in new ways. Start by scanning your room, just that is a very exciting project!

Resources

Here are the links to the TouchDesigner documentation regarding ZED TOP, CHOP and SOP:

TOP: https://docs.derivative.ca/ZED_TOP

CHOP: https://docs.derivative.ca/ZED_CHOP

SOP: https://docs.derivative.ca/ZED_SOP

Also, if this is a topic of your interest, it might come in handy to have a look at the official ZED documentation:

https://www.stereolabs.com/docs

Wrap Up

It’s clear that ZED cameras offer a world of possibilities in TouchDesigner. From interactive art to augmented reality, there’s much to learn and create. These tools open doors for innovative designs and immersive experiences. As we approach the year’s end, we hope this exploration inspires your projects and paves the way for continued creativity. Wishing you a warm and productive end to this year, and eagerly looking forward to seeing all the amazing work you’ll create in the coming year. Happy holidays!