With successes like Oculus Rift and Sony's Playstation Virtual Reality, purveyors of VR and AR systems don't need to hype the technology anymore. They're hard at work making VR and AR systems practical. This Special Project investigates the technological status of VR/AR, and the types of uses they’re being applied to.
By Gina Roos, editor-in-chief
Designers of virtual reality (VR) systems face a lot of challenges. These range from selecting the right compute power and cooling system to packing all of these components into a small form factor, while keeping costs down. But perhaps the overriding challenge is ergonomics — ensuring that the headsets are comfortable for users. Many of the design tradeoffs are related to the form factor. Why? If the headset is too heavy or gets too hot when running content, users aren’t going to use the device.
While ergonomics is “super critical,” it takes a very unique mix of hardware, software, and technologies, said Prabhu Parthasarathy, VR hardware product manager for Facebook, who works on the Oculus VR products.
“When designing any of our products, we never look at a single component in isolation; rather, we consider the overall product experience that we’re trying to create, including its unique audience and use cases,” he said.
But Parthasarathy admits that the “magic” happens in the software and hardware integration.
“The magic happens in two areas — one is writing the software and experiences that make it work like magic, which is incredibly hard, and the other part is getting the ergonomics right, making all of this fit into form factors that allow you to forget the hardware and live in the virtual world.”
From the beginning
There are several types of VR systems, ranging from PC-based systems in which the headset is tethered to the PC, which does all of the heavy lifting for gaming and graphics, to all-in-one VR systems that run all the software in the headset.
Oculus VR is a good example of a company that has done it all. The company launched in 2012 with the idea to make VR affordable for everyone. Two years later, Facebook acquired the company for $2 billion, even though it had only a headset prototype.
“Facebook saw an opportunity and said, ‘The natural progression of connecting people will eventually get to the point where we would like people to meet in virtual worlds and communities,’” explained Parthasarathy.
The first product was the Oculus Rift, a PC VR system, which allows users to run games developed for VR experiences on the PC. These systems don’t restrict game developers in terms of the size of the game or the types of graphics, and with the help of external sensors, the system can determine where the player is in the real world and translate it into the virtual world.
The Oculus Go followed and was the company’s first untethered all-in-one VR system, wherein the software runs entirely on the headset, but it was limited by three-degrees-of-freedom (DoF) orientation tracking. A three-DoF system can track three axes. In a VR headset, it can detect a person moving left/right and turning his or her head up/down and left/right.
But it doesn’t track for forward and backward motion or crouching versus standing. This is where six DoF comes in, which has a significant impact on the system hardware, software, and cost. It detects positions on six axes — forward/backward, left/right, and up/down. Tracking systems can be complex, requiring additional hardware and software, adding to the cost.
The latest innovation is the Oculus Quest, the company’s first six-DoF all-in-one VR system with touch controllers. This system, bridging the gap between the two previous systems, required tight integration between the hardware and software teams.
Oculus Quest is a six-DoF all-in-one VR system with touch controllers.
Building blocks
Using the Oculus Quest VR system as a design example, Parthasarathy walks us through the key components of a VR system — at a high level.
Like many products, the heart of the computing system is the processor. The engineering team selected a Qualcomm mobile chipset, the Snapdragon 835, for the core of its computing system. The Snapdragon 835 has eight cores with clock speeds up to 2.45 GHz and features an Adreno 540 GPU. Oculus designed the system to optimize the performance of the chip, including a new cooling system, to run the CPU at higher clock rates. More about the cooling system later.
If the processor is at the heart of the system, the pièce de résistance is the tracking system.
To make tracking work, Quest designers developed the Oculus Insight inside-out tracking system, one of the company’s core competencies and biggest areas of innovation. This allows what a user does in the real world to instantaneously happen in the virtual world, thanks to all of the sensors (cameras) in the headset as they detect the controllers as they move.
The system uses a lot more data from a combination of inertial measurement units (IMUs), ultra-wide-angle cameras, and infrared LEDs to track the 6-DoF position of the VR headset and controllers.
It’s a complex system, consisting of hardware components — sensors, IMUs, etc. — complicated software including sensor fusion, and computer-vision algorithms, Parthasarathy said.
At the heart of Oculus Insight’s inside-out tracking is simultaneous localization and mapping, or SLAM, which uses computer-vision algorithms to “fuse” incoming data from multiple sensors to fix the position of an object within a digital map that is constantly being updated. It also uses other sensors such as acceleration and velocity data from IMUs in the headset and controllers, which are processed in real time on the mobile chipset.
Oculus Insight consists of four ultra-wide-angle sensors to analyze the environment outside the headset and computer-vision algorithms to track exact positioning in real time. This system tracks a person’s full range of movement and pinpoints the location of the two handheld controllers and headset.
“Insight uses information from these sensors to create a 3D map of your environment in order to keep you safely within the bounds of your play space while translating your movements precisely into VR, resulting in an incredibly immersive experience,” said Parthasarathy.
Digital-signal-processing optimization includes asynchronous map updating, which allows the system to update maps based on changes in the user’s environment in the background. The IMU runs on its own with the output data stored in a memory buffer to minimize system latency.
The Oculus Insight processes multiple threads of data at once in real time.
Next comes the viewing system, or the display and optics. While Oculus uses LCD technology for some of its products, the company selected OLED technology for the Quest. The OLED display provides a 72-Hz refresh rate and a resolution of 1,600 × 1,440 per eye. Quest also incorporates a lens spacing adjustment for visual comfort.
“One of the key things about making a compelling VR experience [is] the lenses,” said Parthasarathy. “These [Fresnel] lenses are extremely special and custom-made for the size and dimensions of the headset and for the kinds of experiences you want to unlock with the headset.
“When thinking about the right display for Quest, a wireless headset, we needed something that could provide great visuals at relatively low power, and an OLED screen combined with our best-in-class lenses — the same ones we used in Oculus Go — provided the right balance,” he added.
Then there is the wireless component. Because Oculus Quest is a standalone system, it uses Wi-Fi. But that’s not all. There are other communications components required, in particular, for the two hand controllers that need to communicate with the headset in almost near-real time.
The real-time part is the big challenge, reducing the latency between what a user does in the real world and translating it into the virtual world.
“The magic of VR is, as I’m moving my hand in the real world, we need that to almost happen instantaneously in the virtual world,” said Parthasarathy. “If there is a lag between what you do in the real world and what you see in the virtual world, that experience will be jarring enough that you are sometimes put off by it.”
To reduce the latency, Quest designers developed their own protocol to communicate between the controllers and headset. As a comparison, Bluetooth Low Energy (BLE), which is one of the lowest-latency protocols, runs at about 7.5 milliseconds, said Parthasarathy. “The communication latency between the headset and the controllers is about 2.5 milliseconds, which is fantastic.”
Inside the Oculus Quest.
Next comes power. One of the key things about VR is to give people experiences that the system does not inhibit, and that means providing battery life that lasts long enough for them to do the kind of things they want to do, said Parthasarathy.
But at the same time, the idea is not to just put a large battery in the headset, he added. “You have to use a battery large enough to accommodate these experiences but also not make the headset very uncomfortable and heavy.”
Not going into specifics, Parthasarathy said that the Quest uses custom battery packs. Though the technology — rechargeable batteries — is nothing unique, the battery had to be a form factor that fit into the headset.
According to documentation, the Quest uses a 3,648-mAh rechargeable lithium-ion battery pack with a 14-Wh rating. The double-cell battery has a nominal voltage of 3.6 V and weighs about 70 grams.
While Parthasarathy said that Oculus primarily uses off-the-shelf technologies, it often requires tight collaboration between its designers and the component suppliers. Whether it’s the battery or display or other components, there is usually a lot of custom work that happens to meet their requirements.
A lot of the components used for the tracking system, as an example, are off-the-shelf parts like the infrared LEDs. But in other cases, parts like the IMUs require some tweaking.
“We use IMUs that are generally available, but our requirements for tracking are so tight that sometimes we go to a vendor and say, ‘Ninety percent of your specs meet our requirements, but can you make the other 10% a little tighter to meet our unique system requirements?’” said Parthasarathy. “We don’t want to be the business of spinning custom hardware for everything, so we try to leverage off-the-shelf components.
“There are very few components on this headset where we just went to a supplier and said, ‘Give us a number of units of this’ and we just integrated them into our system,” he said. “There is often very close collaboration — like for the optics and display — with suppliers.”
Another example is system cooling, a big design challenge in these systems. Oculus Quest uses a combination of fans, heat sinks, and very “novel architectures,” which allows the heat to dissipate and keeps the device from running hot.
It uses an active, fan-based cooling system to regulate temperature, which allows Quest to run at much higher clock rates for sustained periods, so it can get more power out of the Qualcomm Snapdragon 835 SoC. In addition to active hybrid fan technology, the system also includes a heat pipe and custom-designed heat-dissipation paths.
Once the hardware requirements are met, pushing innovation in software is a big pillar for Oculus products. The way to power new experiences is to get to a point where the broad performance metrics are met by the hardware and then push the boundary using software, said Parthasarathy.
“Software innovation allows us to push our hardware to its max potential while continuing to unlock better and better experiences for people over time,” he said. “This is a big area of focus for us.”
One example is the Passthrough+ feature originally launched on Rift S that gives people a stereo-correct, real-time view of their surroundings while in VR (e.g., the ability to “look through” the sensors and see what’s around you while still wearing the headset), said Parthasarathy.
“Making this possible on Quest required advancements in high-performance image processing and 3D computation, and now that it’s launched, Passthrough+ turns on when you step outside of your play space to ensure you can easily find your way back.”
Tying it all together
Ergonomics is a big design concept and, in many cases, is related to the selection of components for a VR system. Everything is a balancing act to achieve the best design, and all of the key components that make up a VR system are design tradeoffs.
It’s all about finding the right balance between the feeling of being completely immersed in the experience and comfortable but also having the right kind of hardware, so the tracking is perfect, along with good battery life and a great display, said Parthasarathy.
Think about it, he said: “Would you like to have a headset that fits on your head or would you like to wear something that feels like your sunglasses but still gives you the same kind of immersion? Everybody wants the sunglasses form factor. The challenge, unfortunately, with the sunglasses form factor is you can’t quite fit in the kind of processing, the battery, or rich optical and display technologies that you want while also making it comfortable enough.
“Ergonomics by itself is not the biggest challenge,” said Parthasarathy. “To provide that compelling experience, you must take everything about the system into account.
“If I want to just improve ergonomics, that’s a very easy one-sided lever that I can press. But that will come with major compromises on other aspects of what we want to do, so everything is a tradeoff.”
VR started with everything residing on a PC, and now, designers are trying to get similar experiences without relying on the horsepower of the PC and graphics card or large cooling system.
“In order to unlock similar experiences on an all-in-one and standalone VR headset, it requires a lot of work and very unique technologies, like fixed foveated rendering, where we can offload a lot of our work very smartly, so we can give as much of the GPU as possible to the developers so they can make great games,” said Parthasarathy.
“If you talk to any systems engineer, they will tell you there is a massive tradeoff exercise between cooling and compute and frame rate and the frequency at which you’re running stuff,” he said. “There is so much happening here, and we are chipping away at it.”
Now, Oculus is working on pushing the graphics and gaming experiences that people are familiar with on the PC. One of the company’s recent innovations is eliminating the controllers in the system. At the end of last year, Oculus developed an SDK that allows VR developers to build experiences that lets players use their own hands without controllers or other peripheral devices.
Quest’s computer vision team used deep learning to understand the position of the player’s fingers using just the monochrome cameras on Quest. Oculus explained that the technology creates a set of 3D points to accurately represent your hand and finger movement in VR.
Hand tracking is one area where software makes a big difference. “Through software innovation, we can unlock entirely new modes of input and interaction without requiring any new hardware,” Parthasarathy said. “This was a big challenge to enable on a mobile chipset.
“To make it possible, our computer vision team developed a new method of applying deep learning to understand the position of your fingers using just the monochrome cameras already built into Quest. No active depth-sensing cameras, additional sensors, or extra processors are needed.
“Instead, deep learning combines with model-based tracking to predict the location of a user’s hands and points on the hands, then reconstructs the “pose” of a user’s hands and fingers in a 3D model,” he said. “And this is all done on a mobile processor, without compromising resources we’ve dedicated to user applications.”
Over the long term, there are a lot of design challenges that VR designers still face, such as making smaller form factors and more comfortable headsets while still providing the same kind of VR experiences, as well as longer battery life. And some of this is driven by the customer — like a reasonable price point of about $400 to $500.
“We are trying to find a sweet spot of continually improving the user experience, giving customers what they want, while pushing the boundaries of technology and making it more accessible,” said Parthasarathy. “I can solve these problems with a $2,500 or $3,000 headset, but that’s not the intention. We want to keep it accessible to most people.”
Disclaimer: I purchased an Oculus Quest for my 60-year-old husband for Christmas, but he still hasn’t opened it.
Articles in this Special Project:
Reality on Display: VR, AR, and MR
By Brian Santo
Introduction to the VR Special Project: With successes like Oculus Rift and Sony's Playstation Virtual Reality, purveyors of VR and AR systems don't need to hype the technology anymore. They're hard at work making VR and AR systems practical.
AR is Propelling Space Manufacturing
By George Leopold
Lockheed Martin has leveraged augmented, mixed reality to reduce touch labor for the Orion spacecraft.
AR/VR/XR – Less Hype, More Substance at CES 2020
By Kevin Krewell, Tirias Research
For many years, the biggest problem with VR has been content, but we are now finally starting to see content that is really driving VR sales.
Kemet’s CTO talks virtual reality
By Gina Roos
Kemet’s deep dive into haptic actuator technologies for VR and wearables began with its collaboration, and subsequent acquisition, of Novasentis.
Advertisement
Learn more about Electronic Products Magazine
