[Editor note from Elburz: Hi all! Today we have special guest Ben Benjamin diving deep into how to remotely control your TouchDesigner installation using Websockets and ngrok. This is a cool topic that a lot of people have been interested in and we’ve talked about recently. We’re glad and appreciative that Ben was willing to bring his web development background to his current TouchDesigner practice and make this great reference for us all. Enjoy!]
Have you ever thought it would be cool to control your TD projects from the comfort of a smartphone? Not just your phone, but any phone located anywhere in the world? We are going to demonstrate how, using WebSockets and a free tool called ngrok, we can interact with our TouchDesigner projects from any device with web access — let’s dive in:
Setting Up Our Project: A Simple Paintbrush App
Before we get into our web setup, let’s get our network running with local data. Our paintbrush will use a standard rendering pipeline with some additional feedback to preserve our brushstrokes. We’ll begin by dropping in a CircleSOP for our brush, wire it to a GeometryCOMP (named “player”), attach a ConstantMAT, and add a camera and a render. Under the camera’s View settings we’ll set our projection parameter to “Orthographic”, and we’ll give our render a square resolution of 1280×1280.
Next we’ll add a ContainerCOMP named “trackpad” with size of 1000×1000, plus a PanelCHOP to drive the position of our brush. We’ll also normalize our uv values from -1 to 1 with a MathCHOP.
And finally, we connect our render to a FeedbackTOP, and composite using the Over operation. At this point we’re able to move our brush around by making our trackpad panel viewer active and dragging our cursor around it.
So beautiful ;___; — jk, let’s add some spice! I colored mine using noise:
- TrailCHOP monitors changes in our trackpad UV (set Grow Length to “On” and Capture to “Only When Input Cooks”)
- MathCHOP controls the rate of change (set To Range to [0, 0.01])
- AnalyzeCHOP with Function set to Sum
- NoiseTOP with Monochrome “Off”, Resolution 1×1, and Transform Y/Z referencing our values
- Set noise as our material’s color map
And that’s it! Now without further ado, let’s…
Put it on the network!
| NOTE: In order for this to work, your computer needs to allow TouchDesigner to bypass any active firewalls. You may be fine depending on what permissions you’ve already assigned, but here are Windows instructions just in case. |
This is the part you came here for — let’s make this controllable from your phone! But how? We’re going to use a recent TouchDesigner addition, the WebServerDAT. Drop one into the network and open up them callbacks:
Many callbacks are written for us by default — hooray! All of these methods are useful for larger projects; for the scope of this tutorial, we only care about onHTTPRequest and onWebSocketReceiveText.
Quick Primer on using WebSockets / Servers with TouchDesigner
At a fundamental level, web servers do two things: they listen for requests and they serve responses. When you visit http://example.com, your browser pings the example.com server by saying, “Hey, can I please view whatever data is located at the URL example.com?” From here, the server can send back a response with a status code: e.g, “No you’re not allowed! [403]”, or “There’s nothing here! [404]”, or “Yes you can! [200]” along with whatever data we originally requested.
Let’s look at our default WebServerDAT onHTTPRequest callback: we set our response status code to 200 — meaning any incoming request is valid or ‘OK’ by default — then we send back some data in the form of an HTML string which includes our WebServerDAT’s name. Sure enough, if you visit http://localhost:9980 in a local web browser, you should see “TouchDesigner: {webServerDAT.name}”
If you’re wondering why we visit localhost:9980, it’s because 9980 is the default port specified in our WebServerDAT. Feel free to change this parameter if you have something else running on 9980! NOTE: If you just updated your firewall settings, you may need to restart TouchDesigner to allow web traffic
Creating A Web-Based Interface
Let’s start by changing the response to include our front-end interface. Create a TextDAT named clientHtml and paste in the following code:
[Editor note from Elburz: If you see the line wrapping happening below, you can copy and paste the full code blocks below into a text editor on your computer and they will be copied correctly!]
<!DOCTYPE html>
<html>
<head>
<title>WS Paint</title>
</head>
<body>
<button id="wipe">Wipe Canvas</button>
<script>
(() => {
// Initialize a new TCP connection on port 9980
const socket = new WebSocket("ws://localhost:9980")
const wipe = document.querySelector("#wipe")
wipe.onclick = () =>
// Send a control message!
socket.send(JSON.stringify({ type: "ctrl:wipe" }))
})()
</script>
</body>
</html>
Don’t worry if you’re new to JavaScript, we won’t go through everything line-by-line in this tutorial — for now we’re only concerned with setting up a remote GUI for our project! Once you’ve copied over the above code, replace the contents of your webserver_callbacks DAT with the following:
import json
def onHTTPRequest(webServerDAT, request, response):
response['statusCode'] = 200 # OK
response['statusReason'] = 'OK'
response['data'] = op('clientHtml').text
return response
def onWebSocketReceiveText(webServerDAT, client, data):
print('Received message: ' + data)
event = json.loads(data)
if event['type'] == 'ctrl:wipe':
op('trackpad').panel.u = op('cam1').par.orthowidth #hide cursor
op('feedback1').par.resetpulse.pulse()
returnIn our onHTTPRequest method, we send the contents of our clientHtml TextDAT to any user who requests it. In the onWebSocketReceiveText method, we await the client (i.e the user) to send us events of type “ctrl:wipe”, at which point we reset our canvas:
Fun! Now let’s level up with a more fleshed out web interface. Delete everything in the clientHtml TextDAT and replace it with the complete code linked here (also listed at the end of this post). Then replace the contents of your web server callbacks DAT with the following:
import json
from datetime import datetime
trackpad = op('trackpad')
feedback = op('feedback1')
def onHTTPRequest(webServerDAT, request, response):
response['statusCode'] = 200 # OK
response['statusReason'] = 'OK'
response['data'] = op('clientHtml').text
return response
def onWebSocketReceiveText(webServerDAT, client, data):
timestamp = datetime.now().strftime('%H:%M:%S.%f')[:-4]
event = json.loads(data)
if event['type'] == 'player:trackpad':
trackpad.panel.u = event['u']
trackpad.panel.v = event['v']
webServerDAT.webSocketSendText(
client,
'{}: {}, {}'.format(timestamp, event['u'], event['v'])
)
if event['type'] == 'ctrl:wipe':
trackpad.panel.u = op('cam1').par.orthowidth
feedback.par.resetpulse.pulse()
return
And we’re done!
Make it Public: ngrok
Our localhost URL isn’t accessible to devices outside our local network — we need to somehow connect to our server via a public-facing URL. Here we can leverage ngrok, a powerful little tool which may make our wildest dreams come true…
Steps
1) Download ngrok — follow instructions for your operating system at ngrok.io. (Yes you have to make an account, but the free tier gives us everything we need to get moving!)
2) Once you’ve extracted the downloaded zip into a folder of your choosing, open that location in your preferred CLI (e.g Command Prompt on Windows, Terminal on Mac). If you’re unfamiliar with command line interfaces, you’ll be using the `cd` command (Instructions)
3) Once your terminal is in the same location as your unzipped ngrok download from Step 1, run ./ngrok authtoken YOUR_AUTH_TOKEN_HERE — you can find your auth token on your ngrok dashboard
4) Now that we’re authenticated we run our final magical command: ./ngrok http 9980 which will bind a new ngrok instance to port 9980. You should see a screen that looks like this:
5) See the highlighted URL? A new one is randomly generated each time you boot up ngrok. Copy and paste the URL into your browser and see we’re now being redirected to our TouchDesigner web server!
6) Take the highlighted [id].ngrok.io subdomain from your instance and swap it into our remote.html on the line that reads const url = "localhost:9980". Remember to refresh your browser to load any updates to the client!
While ngrok won’t let us keep a consistent subdomain unless we’re on one of the paid tiers, we’re fine with the free tier so long as we remember to repeat Steps 4-6 whenever we restart our app. And as long as ngrok is running, your unique subdomain will remain valid until you terminate the ngrok process.
And now(!!!!) if you visit [id].ngrok.io on your smartphone or any device…
Thanks ngrok <3 you’re the hero we deserve!
Conclusions and Next Steps
In this tutorial, we created a simple MS Paint-style scene in TouchDesigner, powered it with data transmitted over WebSockets from a custom web interface, and made it publicly accessible through the magic of ngrok. Here are some fun things we could build on top of our work here:
- Allow 2 or more concurrent players, each with their own unique paintbrush
- Control aspects of the player’s brush from our web interface, such as color, size, etc.
- Mask our ngrok URL with a more permanent domain (e.g example.com) with DNS
The world is yours! Just remember that when you expose a local server to the public you’re allowing web traffic to interface directly with your computer. Great power == great responsibility ????
References
Final clientHtml:
<!DOCTYPE html>
<html>
<head>
<title>TD Remote</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<style>
body { margin: 0; padding-bottom: 2em; overflow: hidden; }
#ctrl_panel { display: flex; justify-content: space-between; align-items: center; max-width: 600px; margin: 0 auto; }
#event_log { text-align: center; }
#trackpad {
width: 100%; max-width: 600px; height: 100vw; max-height: 600px;
box-sizing: border-box; margin: 0 auto; background-color: #c8c8c8;
position: relative; display: flex; color: #a2a2a2; }
#trackpad:before {
content: attr(data-label); margin: auto; font-size: 12vw;
font-family: monospace; }
@media screen and (min-width:600px) {
#trackpad:before { font-size: 5em; } }
@media screen and (max-height:600px) {
body { overflow: scroll; } }
</style>
</head>
<body>
<div id="trackpad" data-label="TRACKPAD"></div>
<div id="ctrl_panel">
<button id="wipe">Wipe Canvas</button>
<span id="event_log"></span>
</div>
<script>
(() => {
// This code should execute on all modern browsers
// To support legacy browsers i.e IE <= 11, see
// https://babeljs.io/en/repl
/****************** Here ******************/
const url = "localhost:9980" // replace with ngrok url
/****************************************/
const socket = new WebSocket(`ws://${url}`)
const trackpad = document.querySelector("#trackpad")
const wipe = document.querySelector("#wipe")
const log = document.querySelector("#event_log")
/**** Post messages to the event log ****/
socket.onmessage = (event) => log.innerHTML = event.data
/***** Trackpad Events *****/
trackpad.ontouchmove = (e) => handleTrackpad(e.targetTouches[0], e.target)
trackpad.ontouchstart = (e) => {
e.preventDefault()
handleTrackpad(e.targetTouches[0], e.target) }
trackpad.onmousedown = (e) => {
const handler = (e) => handleTrackpad(e, e.target)
window.addEventListener("mousemove", handler)
window.onmouseup = () => window.removeEventListener("mousemove", handler)
handleTrackpad(e, e.target) }
function handleTrackpad(user, target) {
const { x, y, width, height, left, top } = target.getBoundingClientRect()
const { clientX, clientY } = user
const { u, v } = formatUV([
(clientX - (x||left)) / width, // left / top supports legacy Edge
(clientY - (y||top)) / height, ])
const json = JSON.stringify({ type: "player:trackpad", u, v })
socket.send(json) }
/**** Control Events ****/
wipe.onclick = (e) => socket.send(JSON.stringify({ type: "ctrl:wipe" }))
/**** Helpers ****/
function formatUV(uv) {
// TouchDesigner expects panel values to increase from the bottom left,
// whereas web events read from the top left, so we invert our v
return {
u: uv[0].toFixed(2), // Round for precision
v: (1 - uv[1]).toFixed(2), } }
})()
</script>
</body>
</html>Wrap up & Thanks
Hi all! Elburz here again. Let me personally thank Ben on behalf of ourselves and the community for making such a great tutorial. I think web interfaces are becoming more and more popular and using the techniques Ben outlined to create a public facing control panel for your TouchDesigner installation easily could be a game changer for your workflows. If you’d like to learn more about Ben and his work, you can hit him up on his website below:
