By Brian Santo, Contributing Writer
Researchers at Disney have devised a virtual reality (VR) system that lets a user play catch with another person using a real ball. But this is not the more well-known augmented reality (AR). The ability to detect a real-world object while still immersed in a virtual world is sort of the opposite of AR. Electronic Products is dubbing this subcategory of VR “augmented virtuality” (AV).
Augmented reality (AR) provides a virtual overlay on reality, usually using transparent glasses. Imagine attending a sports stadium and being able to watch the game, yet still get stats or video replays presented in a corner of your vision. Alternatively, imagine how much easier it might be to put together an Ikea Brimnes daybed if you can refer to an animated overlay that matches the parts in your hand exactly, showing you precisely how they fit together.
The reverse is also possible: overlaying a real-world image onto the virtual world. Creating digital images of real-world objects and inserting them in a virtual world in real time has been accomplished before. Intel, for example, has an AV component in its VR technology development program, which it calls Project Alloy . At the company’s annual developers conference last summer, CEO Brian Kzranich participated in an on-stage demonstration in which an Intel employee was immersed in a VR world; the view from the VR headset was doubled on a flat screen above the stage. The employee, blind to the real world, wandered perilously close to Kzranich, at which point the VR system imported real-time video of Krzanich (frantically making warding-off motions) into the VR world. What looked like an inevitable collision was avoided.
That was an impressive demonstration, but Disney has significantly upped the ante on it with its ball-catching demo.
“It takes acute visual sensing, prediction, temporal planning, and refined motor skills…,” the researchers write in their paper describing the system. And that’s just for the human doing the catching. The VR system has to take into such account such factors as “system latency, rendering of depth cues, frame rate, tracking precision, and registration,” which affect performance.
What the researchers created was a combination tracking-and-rendering system that allowed a user in the VR space to see a virtual ball that accurately portrayed a real ball’s motion. The user could also see both a projected trajectory of where that ball was going as well as a suggested target point for placing their hand to catch that ball. With a VR representation of their real hand’s movement added to the mix, the system essentially showed the user how to catch the real ball that was coming at them while remaining in the virtual space.
Disney’s video tracking system captures video at 120 frames per second (fps), almost five times the 24 fps used to film most motion pictures. The display is an Oculus CV1, using the Unity 3D game engine on a Windows 10×64 system based on an Intel Xeon chip backed by an Nvidia GeForce graphics processor. The researchers highlight that they are using a relatively new mathematical approach, called Unscented Kalman Filtering (developed around the late 1990s) to perform predictive estimates of the ball’s trajectory.
Even with all of that, the researchers had to scale back the rendered environment to the absolute basics (wall, floor, baseball, flat paddles in place of hands) to minimize both the rendering frame rate (140−150 fps) and system latency — and the Disney crew reported that there were still latency lags.
Disney combined three visualizations, including rendering a matching virtual ball, the predicted trajectory of the ball, and a target catching point lying on the predicted trajectory. This video from Disney shows the differences among the three, which include changes in how the catcher catches the ball (a result that the researchers apparently did not anticipate).
As extraordinary as Disney’s ball-catching demonstration is, it’s still a rudimentary system that can and will be refined. The Disney experiments were with underhand tosses. No one would be insane enough to put on a headset and try to catch an Aroldis Chapman fastball.
Given time, though, that might become possible.
Learn more about Electronic Products Magazine