In this tutorial we will show how to integrate multiple WebSocket connections to use in interactive projects in TouchDesigner.
In this post we learned about WebSocket and its importance in the development of immersive and interactive environments.
Today we will learn how to create a custom web interface and use it to manipulate data in real time in TouchDesigner. And, most, of all, multiple users will be able to interact simultaneously with our lovely installations. I am guessing you are already planning how to use it. So let’s go!
Just to sum up, WebSocket is is a bidirectional, full-duplex protocol for client-server communication over a single TCP connection.
The project is split in three parts:
Create a WebSocket Server
The first thing we need to do is to establish a WebSocket server, that will be in charge of the bi-directional data communication.
There are plenty of options to do this. I started from the precious work of Torin Blankensmith, he published a dedicated Youtube playlist as well as a WebSocket node server template.
In order to create the server:
- Login on GitHub or create an account
- Visit the Torin Blankensmith Websocket Node Server Template repository
- Click on Fork to create a copy of the repository
Now we need to create the WebSocket server. There are plenty of available services around the web. I decided to use the cloud application platform Heroku. So:
- Go to Heroku website and create an account
- Click on Create new app
- Choose a name for your app and select region
- Click on Create app
In the next screen, in the deployment method section, click on GitHub, select the repository fork you just created on GitHub and click Connect.
Congrats, we just created our WebSocket server! Now let’s go to the UI side.
Get Our 7 Core TouchDesigner Templates, FREE
We’re making our 7 core project file templates available – for free.
These templates shed light into the most useful and sometimes obtuse features of TouchDesigner.
They’re designed to be immediately applicable for the complete TouchDesigner beginner, while also providing inspiration for the advanced user.
Create the UI
For this project, we want to be able to control parameters in TouchDesigner via a custom web interface, for example a joystick. The interface data, such as X and Y coordinates, speed and angle, will be streamed to TouchDesigner via WebSocket in real time.
We created a Pen on CodePen that allows us to code and see the result directly in the browser. We started from this code example and customized it. For sake of speed, we put the Javascript script into the HTML code, but of course you can create a separate JS file and recall it via script in the HTML code.
Let’s have a look at the code.
<!DOCTYPE html>
<html>
<head>
<title>Mousebot</title>
<meta name="viewport" content="user-scalable=no">
</head>
<body style="position: fixed; font-family: 'Gill Sans', 'Gill Sans MT', Calibri, 'Trebuchet MS', sans-serif ; color:rgb(128, 128, 128); font-size: large;">
<h2 style="text-align:center">Joystick</h2>
<p style="text-align: center;">
X: <span id="x_coordinate"></span>
Y: <span id="y_coordinate"></span>
Speed: <span id="speed"></span> %
Angle: <span id="angle"></span>
</p>
<canvas id="canvas" name="game"></canvas>
<script>
var connection = new WebSocket('wss://super-synth-63d47bc15387.herokuapp.com');
var sliderId = Math.floor(Math.random() * 1000);
var lastDataSentTime = Date.now();
const timeoutDuration = 10000; // 10 secondi
connection.onopen = function() {
connection.send(JSON.stringify({
type: 'connect',
sliderId: sliderId
}));
};
connection.onerror = function(error) {
console.log('WebSocket Error', error);
alert('WebSocket Error', error);
};
connection.onmessage = function(e) {
console.log('Server:', e.data);
};
function send(x, y, speed, angle) {
var data = {
"x": x,
"y": y,
"speed": speed,
"angle": angle,
"sliderId": sliderId
};
data = JSON.stringify(data);
console.log(data);
connection.send(data);
lastDataSentTime = Date.now(); // Update the last data submission time
}
function checkTimeout() {
const currentTime = Date.now();
if (currentTime - lastDataSentTime > timeoutDuration) {
// It has been more than 10 seconds since the last data submission
sendDeleteCommand();
lastDataSentTime = Date.now(); // Timer reset
}
}
function sendDeleteCommand() {
var data = {
"type": "delete",
"sliderId": sliderId
};
data = JSON.stringify(data);
console.log('Sending delete command:', data);
connection.send(data);
}
setInterval(checkTimeout, 1000); // Check every second
var canvas, ctx, width, height, radius, x_orig, y_orig;
window.addEventListener('load', () => {
canvas = document.getElementById('canvas');
ctx = canvas.getContext('2d');
resize();
document.addEventListener('mousedown', startDrawing);
document.addEventListener('mouseup', stopDrawing);
document.addEventListener('mousemove', Draw);
document.addEventListener('touchstart', startDrawing);
document.addEventListener('touchend', stopDrawing);
document.addEventListener('touchcancel', stopDrawing);
document.addEventListener('touchmove', Draw);
window.addEventListener('resize', resize);
document.getElementById("x_coordinate").innerText = 0;
document.getElementById("y_coordinate").innerText = 0;
document.getElementById("speed").innerText = 0;
document.getElementById("angle").innerText = 0;
});
function resize() {
width = window.innerWidth;
radius = 170;
height = radius * 3.5;
ctx.canvas.width = width;
ctx.canvas.height = height;
background();
joystick(width / 2, height / 3);
}
function background() {
x_orig = width / 2;
y_orig = height / 3;
ctx.beginPath();
ctx.arc(x_orig, y_orig, radius + 20, 0, Math.PI * 2, true);
ctx.fillStyle = '#09803e';
ctx.fill();
}
function joystick(width, height) {
ctx.beginPath();
ctx.arc(width, height, radius / 4, 0, Math.PI * 2, true);
ctx.fillStyle = '#241e1e';
ctx.fill();
ctx.strokeStyle = '#6b5e5e';
ctx.lineWidth = 8;
ctx.stroke();
}
let coord = {
x: 0,
y: 0
};
let paint = false;
function getPosition(event) {
var mouse_x = event.clientX || event.touches[0].clientX;
var mouse_y = event.clientY || event.touches[0].clientY;
coord.x = mouse_x - canvas.offsetLeft;
coord.y = mouse_y - canvas.offsetTop;
}
function is_it_in_the_circle() {
var current_radius = Math.sqrt(Math.pow(coord.x - x_orig, 2) + Math.pow(coord.y - y_orig, 2));
return radius >= current_radius;
}
function startDrawing(event) {
paint = true;
getPosition(event);
if (is_it_in_the_circle()) {
ctx.clearRect(0, 0, canvas.width, canvas.height);
background();
joystick(coord.x, coord.y);
Draw(event);
}
}
function stopDrawing() {
paint = false;
}
function Draw(event) {
if (paint) {
ctx.clearRect(0, 0, canvas.width, canvas.height);
background();
var angle_in_degrees, x, y, speed;
var angle = Math.atan2((coord.y - y_orig), (coord.x - x_orig));
if (Math.sign(angle) == -1) {
angle_in_degrees = Math.round(-angle * 180 / Math.PI);
} else {
angle_in_degrees = Math.round(360 - angle * 180 / Math.PI);
}
if (is_it_in_the_circle()) {
joystick(coord.x, coord.y);
x = coord.x;
y = coord.y;
} else {
x = radius * Math.cos(angle) + x_orig;
y = radius * Math.sin(angle) + y_orig;
joystick(x, y);
}
getPosition(event);
speed = Math.round(100 * Math.sqrt(Math.pow(x - x_orig, 2) + Math.pow(y - y_orig, 2)) / radius);
var x_relative = Math.round(x - x_orig);
var y_relative = Math.round(y - y_orig);
document.getElementById("x_coordinate").innerText = x_relative;
document.getElementById("y_coordinate").innerText = y_relative;
document.getElementById("speed").innerText = speed;
document.getElementById("angle").innerText = angle_in_degrees;
send(x_relative, y_relative, speed, angle_in_degrees);
}
}
</script>
</body>
</html>Let’s analyze it:
- Line 1 to 17: standard code formatting
- Line 19: we establish the connection with the Heroku WebSocket server. Remember to insert the link to your own server
- Line 20 to 22: we create a variable called SliderId, that assigns a random ID for each single connected user. We also define two time variables that will be useful for the final outcome, we will see it later on
- Lines 24 to 29: we open the JSON connection and assign the IDs for each user
- Lines 40 to 52: we create the main variables – X, Y, angle, speed, sliderId – and establish the JSON connection
- Lines 54 to 61: we create a checkTimeout function, that resets time if no more data have been received after ten seconds. The following sendDeleteCommand function erase the ID after the ten seconds set interval
- Lines 75 to 93: we define the UI variables and we create the addEventListener events, in order to send mouse and smartphone touch events such as touchstart, touchend, etc
- From line 95 onwards we find all the functions needed to create the UI in Javascript
OK, now we have our wonderful UI we can interact with via smartphone or mouse.
Now let’s open TouchDesigner and create the patch.
Manage Multiple WebSocket Connections in TouchDesigner
So, here we have the patch:
The first thing we need to to is to create a DAT WebSocket component. In the Connect tab we insert our network address – the Heroku address – and set the network port to 443.
Now we to edit the WebSocket callback tab in order to manage the connection between the WebSocket server, the TouchDesigner and the web interface.
Here is the code:
import json
import time
# Dictionary tracking update times for each sliderId
last_update_times = {}
def onConnect(dat):
print("Connected")
return
def onDisconnect(dat):
print("Disconnected")
return
def onReceiveText(dat, rowIndex, message):
global last_update_times
if message == "ping":
dat.sendText("pong")
return
data = json.loads(message)
message_type = data.get('type', 'update')
if message_type == 'update':
if 'x' in data and 'y' in data and 'sliderId' in data and 'speed' in data and 'angle' in data:
x = data['x']
y = data['y']
speed = data['speed']
angle = data['angle']
sliderId = data['sliderId']
# Update last updated time for the sliderId
last_update_times[sliderId] = time.time()
# Update or add the row in the table
update_table(sliderId, x, y, speed, angle)
elif message_type == 'delete':
sliderId = data.get('sliderId')
if sliderId is not None:
delete_row(sliderId)
def update_table(sliderId, x, y, speed, angle):
table = op('test_test')
# Search for the sliderId inside the table
row_index = None
for i in range(1, table.numRows):
if table[i, 'user'] == str(sliderId):
row_index = i
break
# Update values if the sliderId already exists
if row_index is not None:
table[row_index, 'valuex'] = x
table[row_index, 'valuey'] = y
table[row_index, 'speed'] = speed
table[row_index, 'angle'] = angle
else:
# Add a new row in the table if the sliderId does not exists
table.appendRow([sliderId, x, y, speed, angle])
def delete_row(sliderId):
table = op('test_test')
row_index = None
for i in range(1, table.numRows):
if table[i, 'user'] == str(sliderId):
row_index = i
break
if row_index is not None:
table.deleteRow(row_index)
if sliderId in last_update_times:
del last_update_times[sliderId]
print(f"Deleted row for sliderId: {sliderId}")
def onReceiveBinary(dat, contents):
return
def onReceivePing(dat, contents):
dat.sendPong(contents) # send a reply with same message
return
def onReceivePong(dat, contents):
return
def onMonitorMessage(dat, message):
return
Let’s analyze it:
- First of all we need to import json and time in Python
- Lines 15 to 20: we create a onReceiveText function and a global variable called last_update_times, that we will use to erase table data if the interface does not send data since ten seconds
- Lines 22 and 23: we load the JSON messages
- Lines 25 to 31: we create five JSON updated variables: X and Y coordinates, angle, speed and ID (sliderId)
- Line 34: we create an array of IDs based on the variable last_update_times global and assign it the time.time() variable
- Line 37: we call the update_table function that updates a five columns table (sliderId, x, y, speed and angle)
- Lines 39 to 42 are really important: as we saw in the Javascript code, we count for how many seconds a user does not send data. Here, upon receiving the delete message from the script, the Python function delete the SliderId row that is not more active. This is important because only active users will be able to interact with our project
- Lines 44 to 62: we fill the test_test DAT Table component. The code looks for the sliderIds inside the table and updates the x, y, angle and speed variables for each ID, thus allowing us to parse the right data for the right ID
- Lines 64 to 75: we check if the row associated to the ID is active or not. If it is not active, the row is deleted and only active users are held
As you can see, this DAT WebSocket component is the beating heart of our system.
Looking at the other components of the patch, the flow is really simple. We focused on the overall system and not on the creative side. So, in our patch, users can just move a simple sphere on a circular screen. You will be in charge to turn data into wonderful works!
So, data from the DAT WebSocket component fill the table test_test. We pass the table data through a DAT Evaluate component in order to resize data to fit the screen.
Going back to our web interface, you can notice that the Y values of the joystick are specular, to say negative values are in the upper part of the interface and the positive one are in the lower one. In TouchDesigner we can select the Y column of the table via the DAT Select component and pass it on another DAT Evaluate component, where we multiply data * -1. Then we merge and replace the Y to get a fully functional table.
Finally, we use table data to instance the GEO of a sphere TOP component and that’s all!
The Cherry On Top
As you can guess, a point is missing. If we close and open the TouchDesigner patch, the table will include the data received during the previous session. So our screen will display spheres that are no more associated to our current IDs. How to manage it?
You can create an Execute DAT and include the following code:
# me - this DAT
#
# frame - the current frame
# state - True if the timeline is paused
#
# Make sure the corresponding toggle is enabled in the Execute DAT.
def onStart():
# Delete all rows from the table except the first one
clear_table()
def clear_table():
global table
# Get a reference to the table
table = op('test_test')
# Save the first row of the table
first_row = table.row(0)
# Delete all rows from the table except the first one
for i in range(table.numRows-1, 0, -1):
table.deleteRow(i)
# Set the values of the saved first row
table.row(0)[:] = first_row
return
def onCreate():
return
def onExit():
return
def onFrameStart(frame):
return
def onFrameEnd(frame):
return
def onPlayStateChange(state):
return
def onDeviceChange():
return
def onProjectPreSave():
return
def onProjectPostSave():
returnLet’s analyze it:
- We call a clear_table() function in def onStart():
- Next we create the clear_table() function. The function creates a table global variable, references it to our test_test_table, saves the first row of the table, deletes all rows from the table except the first one and set the values of the saved first row
So, as you will open the patch next time, the table will be clear. That’s nice!
Wrap Up
In this tutorial we created a custom application based on multiple real time WebSocket connections. Now that everything is set up, you can create your custom interface and turn data into lovely elements users can interact with in your immersive installations. The sky is the limit!
