What is Immersive Design?

In the overarching analogy, immersive design functions as the phrases and paragraphs of our essay. Using the building blocks of Immersive Technology, we begin to tell our stories and communicate our ideas using different elements of Immersive Design.

Consider trying to express a feeling about nature in a short phrase. What phrase would you create? There really are an infinite number of possibilities. Two examples I thought of are “I love going out into nature” and “Nature is calming”. These two example use very different words, have different sentence structures, different lengths, and fundamentally express two very different ideas. If we relate these two examples back to our ideas around Interactive Technologies as letters and words, both of these phrases would need a very different combination of Interactive Technologies.

This is a very important point to internalize:

Different ideas and stories will require different kinds of Interactive Technologies.

There aren’t any catch-all sensors that track every kind of movement. There doesn’t exist any single pieces of software that can do every single thing that we dream of. They just don’t exist yet. This is where Immersive Design comes into play.

Immersive Design can be simply thought of as the art of knowing the right tools for a job. This often comes with experience, many hours of research, and many trials and errors.

In the above example we were given free reign to create a phrase using any words we would like. In reality, there are almost always limiting factors to what kinds of Interactive Technologies can be used, whether they be environmental limitations, budgetary limitations, or etc. This is where the true skill of an Immersive Designer is revealed.

A more accurate example of an Immersive Design task would be as follows: Create a compelling phrase that expresses a positive idea or emotion about nature, that must include the words “nature”, “I”, and “waterfalls”, while also excluding the words “love”, “like”, “seeing”. Sounds interesting, right?

Definition

Immersive design can be defined in a variety of ways based on the context. For our purposes, we will define it as such:

The art of storytelling across multiple forms and mediums in both virtual and actual dimensions.

There are quite a few different components to this definition.

The first element is the art of storytelling. When all is said and done, design of any kind is about telling a story, setting a mood, or communicating a message and idea. In the realm of immersive technology and installation art this is no different. Instead of using a paintbrush as a painter would, we’ll use motion sensors. Instead of using wood and concrete as an architect would, we’ll use software and computers. The goal of immersive design is to tell a story with seemingly disparate elements, technologies, and pieces.

The second and third elements are related but will be explained separately. The second element is the spanning of multiple forms and mediums. This refers to ability to take an idea, story, or message, that is often linear, and convey it across a number of different forms and mediums (often nonlinear). This is important because in the realm of interactive technologies, many technologies are only capable of a small subset of actions. More often than not, a single piece of technology will not be able to convey an entire idea let alone a whole story. Knowing when and how to incorporate multiple technologies, asset mediums, sensors, actuators, devices, display types, etc, together to tell a cohesive story or express a full idea is the bread and butter of immersive design.

The third element extends the second element to really emphasize the idea that immersive design isn’t just about interacting with digital media. Immersive design may have started in a realm of interactive web-sites or choose-your-own-adventure films, but it has grown greatly and now both virtual and actual dimensions must be taken into consideration and designed for. Virtual dimensions include standard forms of content such as films, images, sounds, web-sites, apps, as well as anything digital that would be perceived through screens, computers, mobile devices, etc. Actual dimensions are all the elements that exist in the real world. This can range from simple elements such as fabricated framing for displays, physical interfaces to control the piece, archiectural elements in the piece, ambient lighting, temperature, the context in which it is presented, and more.

Examples

With a concrete definition let us examine some examples of immersive design.

Before we dig into too many examples, there is an interesting idea that we should contemplate. I often relate immersive design to scoring a film. A good film score is mostly invisible to the individual experiencing it, but is often times the most memorable part of the film and solifidies the film viewing experience. Similarly, in immersive design, you want to create an experience and narrative so fluid and captivating that the user never stops to consider how sensor X communicated with web service Y to analyze the data and create interesting audio and visual feedback Z. In this sense, good examples of immersive design are, by very nature, “invisible”. With that in mind, let us examing an examples of immersive design.

This example isn’t very artistic, but none-the-less it makes for good case studies: Google Hangouts. This may seem like a strange example, but consider the silly and fun apps integrated into Google Hangouts. In one of those apps, Google Effects, you are able to add silly objects to yourself, such as funny hats, fake beards, glasses, and props that track and follow you in real-time. There are algorithms at work that take your regular camera stream in real-time, analyze it to figure out where all the elements of your face and body are, and then overlay the graphics on your image while continuously recalculating the graphic’s position as you move around the screen. If you haven’t tried this, you can start a Google Hangout on your own and add the application to experiment with it. It is a very simple concept, but it is quite effective.

What is interesting about Google Effects is that it is a perfect example of the invisibility of immersive design. Users are already engaged in Google Hangouts for either business or personal reasons, and with this addition, users are able to quickly have a fun and memorable moment within the Google brand and Hangout product that is extremely easy to use and requires no extra thought on their part, aside from deciding what funny extras they want on themselves. The user never has to consider the camera, sensors, software, cloud services, data transmission, grahpics compositing pipelines, user interface, any of the technical backend, let alone any of the software’s limitations, such as not being able to stack multiple hats.

Don’t let the lack of “artistic depth” and simplicity fool you into thinking the narrative wasn’t successful here. I assure you, the creators’ goals and narrative have been fully accomplished. They were able to give the user a pleasent and memorable moment without side tracking them from their original purpose (unless their purpose was to have a serious business call). The narrative from the user’s perspective is that they’d like to have a fun moment during a video call, and are able to do so by simple clicking on Google Effects, and then playing around with fun addons. This is very similar to the concept used in most retail installations, in that brands would like to create a short, fun, and memorable moment in their overall experience without detracting from the user’s purchasing abilities.

This leads us to examine what kind of things qualify under the banner of immersive design.

Four areas of immersive design will be investigated in this series. They are:

• Triggers & User Experience
• Real-time & Generative Content
• Social Media Interactions and Content
• Data-driven Displays (or Info-displays)

Triggers are the output of all of our interactive technologies. Sometimes they are simple and sometimes they are complex, but more often then not, they need to be used and chained together to create a meaningful interaction. What kind of data you decide to source from your interactive technologies, how you chain that data into a cohesive user experience, how that user experience effects the environment or narrative, and how you manage to do that without giving every user a training session is what consitutes Triggers & User Experience.

Real-time & Generative Content is the driving factor of immersive design. The prevalence of fast and affordable computers has allowed for audio and video assets to be created in real-time, which has in turn created a demand for interactive technologies to control these audio and video elements. Knowing what kind of content can be generated in real-time, what kind of things are “generative art”, and how these elements can be affected and grow over time constitutes as Real-time & Generative Content.

Social Media Interactions and Content extends the ideas of Real-time & Generative content by integrating the vastness of all different kinds of social media. Knowing what kind of data, how often that data can be accessed, and how to create Triggers & User Experiences out of seemingly unrelated social data constitutes Social Media Interactions and Content.

Finally, Data-driven Displays (or Info-displays) takes the notion of using social media as a data source and extends it to the myriad data that exists on the Internet. We will explore the use of data sources like weather, traffic, sports, and news, in interactive installations. We will consider how they can be used in entertaining ways, but also how they can be used as valuable sources of information for users and the community.

What’s next?

With both the interactive technology and immersive design defined, next up is definition and examples of technical implementation, which is possibly the most important part since that is where most great ideas fall flat on their face. Soon, we’ll get to exploring TouchDesigner.