Datamoshing in TouchDesigner – Part #1

Datamoshing is a super fun effect to build and has been increasingly finding more creative usages both in real-time and rendered media. Implementing it can be a bit tricky though and in this 2-part blog series, we’ll look at a fun way you can build a datamoshing effect in TouchDesigner.

What is Datamoshing?

Datamoshing is a glitch art kind of effect that is created by actually damaging/corrupting/warping frame data to simulate what happens when compression algorithms fail to work properly. This question first came up in The HQ PRO where one of our members was trying to build this kind of effect, and there’s lot of examples of this but this A$AP Mob music video shows it off well:

Specifically near the beginning around 18 seconds and 23 seconds you’ll see effects like this:

You may have seen effects like this before if you had a corrupt digital video file or low powered hardware having a tough time decoding heavily compressed videos. Now it’s a creative effect that has a lot of potential!

How does it work?

Half of the battle of recreating any effect is knowing how it works. Some effects are easier to brute force your way to intuitively, but others require a bit of reading. Datamoshing effects in their original appearance are created by issues between the different types of frames in compressed videos. Not all frames in the video are actually full frames of data. If you’re reading about datamoshing online you’ll often hear about I Frame, P Frame, and B Frames. Without getting into the weeds, the important thing to note is that many compression algorithms try to only hold the amount of data in each frame that has changed since the last frame. That’s why files with not much motion in them (most of the frames are the same) can be compressed to very small sizes while textures like noise or grainy videos (where lots of information per frame is changing) have a tough time being compressed to smaller sizes.

Here are two images that show these concepts at a high level just for visualization purposes. The both show how in many cases a video will actually have different types of frames holding different amounts of information:

Implementation Concepts

Here’s a great primer on how you can implement a datamoshing effect in Unity:

Creating Your Own Datamoshing Effect

While we can’t use everything from the Unity implementation, the author does a great job of walking us through the high level concepts of what needs to be happening. In our implementation we’re going to use a mix of GLSL and some pre-built tools in TouchDesigner to make the magic happen. If you’re new to GLSL, don’t worry, in the next blog post we’ll explain all the lines of code we’ll be writing. So our first step to implementing something like this is to break down their article into some key steps:

• Get/create a set of motion vectors for movement in the entire image
• Displace our colour original image sampling UV position
• Find a way to recycle the pixels on screen / don’t clear screen every frame
• Add a mechanism to blend between unprocessed and processed pixels
• Quantize the sampling blocks of our movement vector to create a blocky effect
• Add some noise per block (what would we do without noise?!)

Then we should be in the ballpark of a cool effect! Now while this may seem a bit pedantic and needlessly “documentation-filling” but there’s an important skill to practice here. Our TouchDesigner community is still comparatively small when compared to Unreal or Unity communities. More often than not, if you’re looking to implement something in TouchDesigner that hasn’t already been explored by other pros, the only resources you’re going to find are going to be made for other platforms. Being able to have the patience and process to take other platforms tutorials and translating them into the steps we need to implement something in TouchDesigner is invaluable. And it starts with simple steps like reading articles and making notes. This is the exact kind of process I follow if I’m confronted with if I have to implement something in TouchDesigner that isn’t already in the collective common knowledge of TouchDesigner developers.

What do we need for datamoshing in TouchDesigner?

The next step of the process is taking our bullet point list of needs and mapping them onto processes or tools we have available in TouchDesigner. Our first bullet was creating/getting a set of motion vectors. If you’ve seen our previous posts/videos, we’ve talked about optical flow before as an easy way to generate motion vectors from videos and camera feeds, and even better is that there is already a fully functional opticalFlow component in the palette we can use:

When it comes to displacing our UV co-ordinates, this will be something we’ll do inside of our shader pretty easily.

When it comes to recycling pixels, there’s a few different ways we can approach this, but likely my mind already starts thinking about using a Feedback TOP and creating a feedback loop between the output of my shader and its input.

Blending between the input and the glitchy effects are something we can do at the output stage of our shader. Similarly, quantizing our sampling blocks can be done with a quick bit of math. And finally adding some noise to the blocks is something we can either replicate from the example blog post or by sampling our own internal noise textures (added creative opportunities!). So even before opening TouchDesigner and worrying too much about the specifics, I’ve already started architecting my implementation and I know it’ll look something like this:

Having this kind of plan of attack is going to make our implementation A LOT simpler next week when we dive into it. Again, while this may seem overly complicated of a process of doing these steps, they truly are what separates people that sett unique trends and the other folks who are repeating commonly implementable techniques on projects. There’s nothing wrong with being either (and in fact I’m on both sides of this regularly!) but I think most folks have more fun making new things.

Wrap up: Datamoshing in TouchDesigner Part #1

In this post, we’ve laid a solid groundwork for implementing a datamoshing effect in TouchDesigner. While this planning work isn’t glamorous, it’s what makes these really tricky concepts much easier to build yourself. Data moshing is an amazingly cool effect and it could become a new main-stay effect like Rutt-Etras. In the next post, we’ll dive into building this in TouchDesigner!