The Interactive & Immersive HQ

Audio Reactive Drawn Content in TouchDesigner

[Editor’s note from Elburz: Hi all! Today we’ve got a great in-depth guest post from Crystal Jow about her process in making this piece of analog-digital hybrid art in TouchDesigner that is audio reactive. She goes through all the nuts and bolts and has kindly provided a TouchDesigner project file you can download and experiment with as well. Everything from the content, she does a lot of great work, and I’d highly suggest checking out her Instagram account by clicking here. Enjoy!]

Do you ever wonder how to get a 2D analog look in TouchDesigner? Or do you wonder how you can make your physical sketches to be audio reactive? Then this how-to is perfect for you.

touchdesigner audio reactive

You can follow along with the project files below:

Preparing the Assets

Start off drawing and painting a frame of the eye open and another frame with the eye closed.

touchdesigner audio reactive

Scan the illustration and open the files in Photoshop. Make the background transparent by double-clicking the layer and adjusting the blending option.

touchdesigner audio reactive

Do the same for the eyes closed file. Save the file as a PNG file.

touchdesigner audio reactive

For more visual variation we will also have the pupil’s eye color be audio-reactive. Mask out the pupil using the Polygonal Lasso Tool and save it as another PNG file.

touchdesigner audio reactive

Eye into TouchDesigner

Open up TouchDesigner and open the assets with a Movie File In TOP. Layer the pupil file over the eye file by using an Over TOP and add a Level TOP after the pupil.

touchdesigner audio reactive

Add a Switch TOP between the open eye and close eye file. Switching the Index between 0 and 1 will make the eye “blink.”

touchdesigner audio reactive

Add a Null TOP after the Switch TOP and name the Null “eye.”

Eye Rays

To understand the technique more can watch Bileam’s tutorial:

Add a Line SOP have Point A be 0.5 and Point B to 3.

touchdesigner audio reactive

Connect a Copy SOP to the Line SOP. Change the number of copies to 20 and set the Rotate rz expression to 360/me.par.ncy

touchdesigner audio reactive

Connect the Copy SOP to a Geometry COMP. Make the rays rotate by putting “absTime.frame*.1”  in the Geometry Rotate rz.

touchdesigner audio reactive

Use a Line MAT for the material. Change the Width Near to 15 and Width Far to 10.

touchdesigner audio reactive

Add a Camera COMP and a Render TOP to the scene. Create a Noise TOP and connect the Render TOP and Noise TOP to a Displace TOP. In the Displace TOP set the Displace Weight to 0.1.

touchdesigner audio reactive

In the Noise TOP set Exponent to 1.15, Amplitude to 0.1, and the Output Resolution to 1920 x 1080

touchdesigner audio reactive

From the Palette add a feedback component after the Displace TOP. You can find the component in the Pallette under Derivative/ ImageFiter.

touchdesigner audio reactive

Color the rays by multiplying a Ramp TOP. Set the ramp as Circular and Output Resolution 1920 x 1080. Change the ramp colors to your desired color.

touchdesigner audio reactive

Add a Background

Add another Ramp TOP. change the Output Resolution to 1920 x 1080 and set Type to Circular. Animate the ramp by adding the expression “absTime.second*.5” in the Phase parameter.

touchdesigner audio reactive

Add a Blur TOP after the Ramp TOP. Set the Pre-shrink 8 and Filter Size 32. Use a Composite TOP to composite the eye, animated rays, and background altogether. Set the Composite Operation to “Over.”

touchdesigner audio reactive

Set-up the Audio

The kick drum from the audio will trigger the different animation elements. Frist put an Audio File In CHOP and connect an Audio Spectrum CHOP right after. Change Audio Spectrum CHOP’s Output Length to “Set Length Manually.”

touchdesigner audio reactive

Add a Trim CHOP after the Audio Spectrum CHOP.Set the Unit Value to “Absolute.”

touchdesigner audio reactive

Look at the Audio Spectrum CHOP for where the kick drum is in the spectrum of frequencies. If you turn on Audio Spectrum TOP viewer active and right-click you can change the Units to “Samples” and Grid Details to “Grid High” to better see the range of the kick drum.

touchdesigner audio reactive

For the eye project, I set the Trim CHOP start at 300 and end 600.

touchdesigner audio reactive

Add an Analyze CHOP after the Trim CHOP. Set the Function as “Maximum.”

touchdesigner audio reactive

Combine the two channels with a Math CHOP. set Combine Channels as “Average.”

touchdesigner audio reactive

Add a Null CHOP after the Math CHOP, rename the null to “AudioFrequency.”

TouchDesigner Set up to Become Audio-Reactive

Make the Eye Blink

Add a Logic CHOP after the AudioFrequency Null. Change the Convert Input setting to “Off when Outside Bounds” and have the bounds to be 1.2 and 2. The bounds is the peak of the frequency.

touchdesigner audio reactive

Add a Lag CHOP after the Logic CHOP and set the Lag to 0 and 0.3.

touchdesigner audio reactive

Add another Math CHOP after the Logic CHOP and change the Integer setting to “Floor.” This data will be used to switch the eye open and closed.

touchdesigner audio reactive

Add a Rename CHOP and rename to eyeSwitch. Add a Null CHOP afterward and reference that value as the Index parameter of the Switch TOP that switches between the eye close and eye open.

touchdesigner audio reactive

Audio-Reactive Eye Scale in TouchDesigner

Create a Math CHOP from AudioFrequency Null. We’re starting a new chain of processing the data that will control the scale of the eye. Change the range of the Math CHOP to 1.2 to 1.6.

touchdesigner audio reactive

Add a Filter CHOP and change the Filter Width setting to 0.38. The filter makes the movement smoother.

touchdesigner audio reactive

Add a Rename CHOP and rename the chain to “eyeScale”. Add a Null CHOP  after the Rename CHOP. Create a Transform TOP after the eye Null TOP. Reference the eyeScale value as Transform TOP Scale sx and sy.

touchdesigner audio reactive

Audio-Reactive Pupil Color

Create a Lag CHOP from the AudioFrequency Null. Change the Lag setting to 0 and 0.3.

touchdesigner audio reactive

Add a Rename CHOP after the Lag CHOP and rename to “eyeColor”. Add a null CHOP after the Rename CHOP. Reference the data value as the High R value in the Level TOP for the pupil.

touchdesigner audio reactive

Post Processing

Post-processing is really what adds the polish to these kinds of things. You’ll see in this project some of the effects I used like adding a glow behind the eye and adding grain to the visual to give a more analog look. You can also add other filtered from the palette that can help improve the quality of the final product.

Takeaway

No need for keyframing! Now you can turn any of your sketches into an audio-reactive masterpiece! You can explore different tracks and see different results you can achieve