Developing Large-Scale TouchDesigner Projects

Developing TouchDesigner projects like a professional, not a pro

This post is going to talk about a few things that seem unimportant or like side worries, but the more and more you work on TouchDesigner projects and the larger the team/project you’re working with/on, you start to weigh things differently.  “Do you remember that one project where the X and Y were all criss-crossing and we couldn’t add feature Z without completely breaking the project?! God that was a disaster, I didn’t know where anything was.” <- Literally said by me maybe a month or two ago! Organization, naming conventions, and TOX files are three things that are boring to think about, but what most high-end developers talk about/laugh about as TouchDesigner projects come and go.

A lot of my notes will be focused on how to benefit the team, which to me is anyone working on or related to the project. If you’re a solo developer, these are still important because it’s the kind of things we look for when hiring and what make us want to rehire someone in the future. It’s not uncommon for us to work with another TouchDesigner developer on a project, for them to do good work but in a messy way, and then for me to just say “Well, it was nice having them but this is a mess and now I have to deal with it, I’d rather just do it all on our own next time even if it takes an extra week…” So take these things to heart because they’re more important than you think for hiring!

Organize your network

If you just did this one thing, you’d level up your skills by 50% as far as I’m concerned. I don’t really care how amazing of a shader you can make or what kind of tricks you can do in TouchDesigner, if your network isn’t clean and organized, I’m going to be frustrated the whole time I’m working with you/your TouchDesigner project. I’ll define the two common problems I see in TouchDesigner projects and their solutions.

Stream-of-features programming

This issue stems from the nature of our work + not starting a project with structure. It’s like stream of conscious writing but for TouchDesigner. Often times, we’re told to do a few things, then we get more instructions and keep building features wherever we happen to be working in the network, then we get even more instructions, then there’s a change in the project for some reason so we copy and paste large parts of the network, then some features get removed and replaced with different version so we bypass some opeartors, etc etc. All the while, you’re just working on your project, adding/deleting/expanding/shrinking networks that directly in front of you and maybe trying to keep related elements on the same networks, but invariable this becomes a disaster. You’ll end up with completely different processes becoming co-dependent on each other. I don’t follow MVC (model view controller) principles by law, but there’s merit in separating functionality (infrastructure, api connections, hardware management), view elements (renders, shaders, etc), and controllers (UI elements, control signals to and from other systems, sensors, etc). No project is really clear cut, but the more clear cut you can make it, the easier it is for other people to quickly jump into your network and continue building elements.

Flat programming

This one is a by-product of the above, but also just something some people seem to like doing. The node-based programs can look really cool if there are hundreds of wires all criss-crossing and overlapping and creating super-highways of complex wires and nodes and blocks…cool! So people just stay on the same network and pan around endlessly, dropping in functionality, features, references, operators, large Text DATs with comments and notes, and all kinds of other stuff. There’s a reason you’ve never seen any of our (nVoid) TouchDesigner project screenshots on any of those “look at this crazy/cool network! wow that’s intense!” posts/forum anywhere. Our projects are so organized and clean that they aren’t cool to look at…they make sense and are simple. There’s never really more than 10-20 operators in a network, because if it starts to go beyond that, we take a step back and check to see if we’re mashing a bunch of features into the same network, at which point we’d probably just make a few sub-components and wire them up/use references to move data between them. What a lot of new developers don’t realize is that rendering large networks takes a lot of your systems resources.  We’re often working over remote connections (like TeamViewer or Anydesk) and having to render less nodes per screen greatly increases remote connection performance as well.

Name your network

This is the easiest starting point for working on large-scale TouchDesigner projects. Name you Container COMPs and Base COMPs holding different elements of the network so that they clearly state what they do or what functionality they might belong to. Easy! The two big reasons for this is that it makes it firstly easier to navigate for your team and the second is that it makes it easier to maintain the project over longer periods. In terms long-term maintenance, you will absolute forget how a network is laid out after about 6 months of not looking at it, I guarantee it. Clear naming conventions can make for quick re-learning of your network for maintenance or upgrades. In the short term, you and your team are working on this project and you want everyone to be able to navigate easily through new functionality and networks with the least amount of questions. Clear naming is the easiest way to add transfer-ability to your TouchDesigner work.

Your paths should be so clear that someone who is only slightly familiar with TouchDesigner should be able to navigate your network and get to general areas of functionality without any previous information about the proejct. Consider the two paths below:

/project1/container3/base3/kinect1

/client_name/external_hardware/kinect_cams/kinect1

Which one is easier to read, understand, and navigate? With that in mind, there is a state of too-much-naming. For example, nested containers don’t need to restate their purpose. The first path below has a bit of unneeded redundancy while the second is clear and concise:

/some_project/external_hardware/external_hardware_kinects/kinect1

/some_project/external_hardware/kinects/kinect1

It only takes a few seconds to name components and create a sense of organization. If you’ve followed the above section on organizing your network, then naming is easy and will make you immediately look like a pro. If you haven’t don’t any kind of organization or structure before hand, then naming will help keep you sane over the life of the project, if nothing else.

Using TOX files (and git) in TouchDesigner Projects

The are two main benefits to using TOX files. The first is that it becomes possible to work with a team on one TouchDesigner project at the same time. This isn’t possible with TOE files because only one person can save the whole project file at a time, so you’ll get merge/sync conflicts on whatever platform is trying to manage the difference of the new TOE files. With TOX files, everyone can work on a different TOX file that can be saved independently and they all get loaded into the same TOE file (that never needs saving). The second benefit is that you’re able to use version control systems like git over the life of the project to manage changes and development of all the TOX files. If you’ve done the above two strategies and used a lot of components that only do a few things and they’re named, then using TOX files is easy (but a little slow). Before proceeding, you should decide how deep you want to go with your TOX file structure. Some people will make every single component a TOX file, and some just make only a handful of TOX files that cover the big feature sets. It’s up to you to decide and experiment to see what works. For bigger teams, more TOX files will be helpful, whereas smaller teams might be fine with only a handful of big TOX files.

To use TOX files: you save each component and subcomponent that you have as a TOX file by right clicking on it, saving it as an external component, and then in the component’s parameters, putting the path to the new TOX file in the “External .tox” parameter on the Common page. As you’re working, you go to your component and right click on the component and re-save the TOX file. This generally works but can become extremely cumbersome. Most people who work a lot with TOX files have internal tools they’ve developed to speed up the process. One of the component on the nVoid nVC2 TouchDesigner Library is a tool that makes it a lot easier to manage TOX files by providing save shortcuts, automatic TOX creation/linking, as well as saved/unsaved status on all TOX files in a project. We use it on every single TouchDesigner project, otherwise I’d get tired of TOX files very quickly!

Real example

Ok, so talk is cheap. With all that in mind, here are some non-nda-breaking screenshots from a recent project that highlight all of the above elements.

 

 

 

 

That’s a quick example from “top” to “bottom” of one feature of a project. Every component that we went into in the sequence above was an external TOX file and we usually use git (GitHub/GitLab) to manage versions, but if it’s a really small project with maybe one or two people on it, then Dropbox also has worked fine for us. Any direction you go in the network for the project above, you’d end up with a very similar looking sequence, more features and stuff until you reach a specific feature at the “bottom”.

Wrap up

In hindsight, this might feel like a boring topic, as it’s not 1600 words worth of “name you components, be organized, and use TOX files” but it is honestly what separates people we always want to work with and people we don’t. It will also help you in the long term in managing and maintaining TouchDesigner projects, especially if you’re not the one who needs to do the hot-fix (common problem if you can’t be on-site and you have to send a subcontractor). Name your comps, be organized, and use TOX files…future-you will thank me!