A new world of gaming: Creating mixed reality

A new world of gaming: Creating mixed reality

Robotics and software enable the creation of entertainment experiences that bridge the virtual and real worlds

Orbotix, Boulder CO
www.gosphero.com

The games we remember from childhood existed entirely in either the real or virtual world. Chess, Monopoly, and even LEGOs are all made up of real-world, physical objects that we can touch. Video games, on the other hand, live in the virtual world — think Halo, Portal, Call of Duty: Black Ops, or even Pong. But today, mixed reality is already a part of our daily lives. Two great examples are the Wii and Kinect. Both use real-world gestures to control virtual objects on the screens of our devices. Even basic tilt games on an iPhone or Android can be thought of as mixed reality.

Mixed reality exists on the spectrum between the virtual and real worlds. At the edges of this spectrum, mixed reality won’t be very noticeable. Done right, it will feel natural and simple to use. That is the goal of creating mixed-reality (MR) games.

Fig. 1: Mixed reality exists on the spectrum between the virtual and real worlds.

What’s in the middle of the spectrum, and is it really that important?

Let’s look at Wii and Kinect. Both live at the right end of the MR continuum, but are clearly integrating real-world interactions into their game play. With Kinect, your physical actions in the real world are connected to your virtual character on the screen; if you jump, he jumps.

At the left end of the MR continuum is the real world with a virtual world overlay. An example is the backup camera on a vehicle. When someone backs up, not only does she see what’s behind the vehicle, but she has guidelines that tell her which direction to travel to avoid objects. The camera can even place red blocks on things that seem like expensive collision points — don’t backup into the red blocks, or it will cost you!

What this means to you

Building MR experiences is technically difficult, and it only gets harder as you move closer to the center of the MR spectrum. Why is it so hard? The challenge falls into two main areas: computational complexity and user acceptance.

Essentially, computational complexity means that computers need to do a ton of processing in order to understand the real world. The real world is a messy place. Detecting edges, people, objects and movement in a variety of lighting conditions, with all kinds of things that can get in the way, requires fast processors and lots of hair-pulling math.

How can you detect a couch if the lights are off or if there is no standard shape for a couch? How can you tell the difference between a ball and a round leg of a table? Not only do you have to understand the real world in computer terms — you have to understand it in real time so it can be used by an application. Lucky for us, in the last few years, consumer grade computers and smartphones have evolved to the point where they can do enough computations to figure this all out. But we still need more power. Today’s technology it is just barely good enough.

The next challenge is designing applications that are meaningful for us — the user. From our perspective, Kinect is very simple — it’s precision is very crude — but technically it is unique. The games and apps have to be designed around the technology and the player has to find that compelling. It would be really cool if the Kinect could detect if I was holding up one finger or two — but it can’t — it doesn’t have that fidelity (yet, anyway). So the games must be designed at a level that works with the technology. It would be great if our cars could back up all by themselves without hitting objects or people — but the technology isn’t quite there yet — so we have yellow lines and red blocks.

So what are we trying to do with Sphero?

Our product, Sphero, is the world’s first robotic ball gaming device that you control with a tilt, touch, or swing from your smartphone or tablet. You can even use Sphero as a controller for on-screen game play. We are trying to move closer to the center of the MR continuum with our game play.

Fig. 2: Sphero, is the world’s first robotic ball gaming device that you control with a tilt, touch, or swing from your smartphone or tablet.

Sphero is a real-world object with a ton of sensors. Sensors produce data that a computer can easily understand. A smartphone can easily process that information and get quite a bit of data on what Sphero is doing. We can then incorporate that information into the game.

Smartphones are also great at virtual-world games — plus they are portable and have a camera. The camera is one important element for Sphero’s MR experiences. When you look at Sphero through the camera, you are creating a new view of the world. The smartphone can place virtual objects in that view while also identifying where Sphero, a real-world object, is in relationship to those objects. This is different than just detecting what is behind a car or if someone is jumping up and down because we can affect the behavior of the real-world object — Sphero. We are placing a virtual obstacle in front of a real-world ball and seeing how it responds.

As an example, we can put up a virtual wall in front of Sphero and he will not be able to drive through it. In the real world, there is no wall, but in the new worldview of MR (and the smartphone’s screen), Sphero conforms to both the rules of the virtual and real worlds. Like Kinect, we have to design game play that works with the capabilities of the technology — but even this basic capability opens up a whole new world of possibilities.

Sphero already has many games and apps that fall along the MR spectrum — some are very simple, like driving Sphero along with a virtual joystick on your device. Others, like Chromo, allow you to control the virtual world from the real. But even in our simplest app, Sphero lives in the MR continuum — engage in the virtual world, play in the real.

The bottom line is these types of games and applications are changing the way we interact with technology. By blending the two worlds, we are opening up a new era of gaming — one that will require more power, speed, and creativity. ■