Guiding Light: What We Learned from Mixed Reality Prototyping

If virtual reality is for escape, then mixed reality is for enchantment. Mixed reality, also known as integrated reality, is comprised of objects from both the physical world and virtual reality. The rules of these new realities have yet to be established and most of the logic that governs these new magical landscapes is still waiting to be discovered. By building mixed reality environments now, we’re able to start to uncover questions and intricacies of this new reality and reveal the unique properties between both its physical and digital components.

At argo, we are exploring the possibilities of mixed reality through projected light. To do this argo creative technologist, Jarrett Webb, is leading a speculative design project called Interactive Light.

Interactive Light is a mixed reality version of the vintage air hockey table game. It uses a mixed reality object — a projected circular light resembling an air hockey puck — the movements of which are controlled by a set of sensors and actuators. Because of the complex interactions between the factors that affect how the light moves and reacts, its movements appear unpredictable giving the impression of a real air hockey puck. Our setup included propping a light projector on a stand and positioning it downward perpendicular to the ceiling so it projected onto a table. Also connected to the stand is a Microsoft Kinect, which is able to detect objects in the projected region. In this case, the projected region was a table..

The projector casts an outline that circumscribes the play area. The puck is also projected creating air hockey via light.

Creating this prototype helped us start to understand what it means to design for light as a material and how it responds to its environment. For starters, we mimicked all the physical elements of air hockey. The pong behaves realistically: its trajectory and spin are aligned with what you would expect from physics and gravity. It’s almost like the real thing.

But there are distinct differences. Light projects indiscriminately, which means that the game requires a clear, smooth surface. Place different objects in projected light and the light will refract or bend to pass through different material. This means the game can be warped by something as simple as an accidental out-of-bound arm, which creates new game play boundaries.

Thus the horizontal projection needs to take into account the surfaces it is projecting on. When horizontal projection is used to create a shared interactive experience, light refraction needs to be at the forefront of designing for that medium. Put another way, were we to project light onto our desks as an additional interactive interface, then we would have to account for all the spaces we don’t want light to project onto: computer, keyboard, scattered notes, forgotten drinks. Knowing when to mask light and how to adapt its trajectory are variables that will require standard design patterns.

We approached Interactive Light wanting to understand how adjusting the angle of the projector changed the light, the environment and our social interaction with it. We were interested in exploring how we interact with objects that were inanimate and artificial, but still responsive to their environments.

Design research at argo begins in the same way. “We design with circuits and code,” Webb says. “They are our material and to design with a material you must first explore its properties and characteristics — find its nature.” Using this think-by-making approach to prototyping, we were able to quickly discern the nuances of the materials we were using and how they impacted aesthetics and usability.

Ultimately, what we learned was that introducing people to mixed reality is, in many ways, a borrowed, social process. From a user experience perspective, we nearly always wanted to grab the puck. It’s a natural instinct. Everything about the puck seems real that while playing you temporarily forget it is not a physical object.

This is the borrowed gameplay behavior carried over from an established, real-world game: air hockey. We elicit our behaviors from these past experiences in the physical world and shift them, over time, into new design paradigms. But the challenge is to not simply imitate the physical world in digital form: it is to create a distinct world with its own advantages. Games like Interactive Light can help us figure out what we expect from mixed reality environments and how we want to interact with them.

While the purpose of Interactive Light was never about creating a game, we could understand what it means to work with light through prototyping and gameplay. Presenting it publicly gave us insight into how people react to an interactive, collaborative mixed reality environment.

Interactive Light is just one example of how we envision think-by-making approaches being applied to test the viability of an idea in the real world. Making a game allows us to play it. Creating something allows us to experience it.

One of our designers, Lisa Woods, gave a talk recently to Girls Who Code. In her talk she said, “People think that the future is a place you go. The future is a place you create.” This feeling of possibility, helping to discover and explore and define what comes next, is the realm of designers in this new mixed reality future we’re entering. We shouldn’t be constrained by what exists. We should create what is next.

A version of this article appeared in HOW magazine on November 17, 2016.

Tiffany Chow is an interaction designer and researcher. She focuses on accessibility, content strategy, and information architecture.

We are a product design firm. We love design – for the technology, for the simple joy of craft, and ultimately for the experiences we create.