Mixed reality: The software is willing, but the hardware is weak

By Richard Quinnell, Editor for Special Projects, Technical

Virtual reality (VR) and its sibling, augmented reality (AR), have long been popular with gaming enthusiasts, but they also have a serious side. At this year’s Hannover Fair, the industrial uses of VR/AR were on prominent display but with one hitch: High-quality application software was plentiful, but the headset hardware for utilizing that software still needs work.

A variety of industrial applications for VR/AR, together known as mixed reality, were on display at the Fair. With VR, engineering collaboration was a major thrust. Companies such as Holo Light, for instance, offered Holo-View software for turning CAD drawings into virtual holograms with which multiple users could interact while wearing VR headsets. Techviz and Virtalis offered collaborative VR that used projectors with users wearing 3D glasses to see the virtual device on any type of stereoscopic display. Both allowed users to manipulate the virtual object, view cross-sections and hidden parts, verify assembly clearances, make measurements, annotate the image, and more.

Another common application for VR involved education and training. Students could interact with virtual machines on their own or in a full simulated environment to practice without risk of injury or equipment damage. Training applications also utilized augmented reality, superimposing instructional diagrams on real objects to guide the student through the steps to be performed, pointing out and labeling the location of key elements, and offering hands-free informative text and videos for student review.

Providing workers with virtual context-aware information display and controls is a prime industrial application of mixed-reality software. (Source: Holo-Light)

Augmented reality saw numerous other factory applications as well. One of the most basic was the use of AR to provide context-aware virtual displays of data, such as temperature, motor speed, power efficiency, and the like. The user only needs to look toward an object to bring up its information in the AR display, hovering around the object like word balloons in comic strips. Some applications also allowed users to invoke menu options or activate basic machine controls in the mixed-reality space simply by directing their gaze toward virtual controls.

Several AR systems aimed at assisting humans in the assembly of complex systems. Rittal, which manufactures enclosure cabinets along with supporting fabrication and assembly tools for populating those cabinets, was showing one such AR system in conjunction with its sister company Eplan. The system aimed to help in cabinet wiring and assembly by showing the assembler both the description of the next wire to install as well as superimposing its connection endpoints and routing path on the real object. This allows users to follow a wiring assembly checklist without needing to look away from the work area.

Augmented reality can help guide workers through operations such as the wiring of an electronics cabinet by pointing out the intended endpoints and routing of the wires involved. (Source: Rittal)

Some AR applications blended multiple objectives, such as collaboration, education, and assembly guidance. Innovae, for example, offered applications that support maintenance and troubleshooting efforts by combining a user’s real-world view with a “digital twin” virtual object and then sharing that combined image with a mentor or expert, who might be far away from the actual machine. This second user could then coach the first on where to look and what to do, pointing out key elements by highlighting them on the digital twin for easy identification. An AI system could also be configured to perform the mentor role, helping a worker troubleshoot by analyzing system interdependencies in the digital twin and pointing out corresponding components for test and evaluation. (For more on digital twins, see “‘Digital Twins’ enable machine simulation & maintenance, Industry 4.0 ” on EP’s sister publication EDN.)

The multiple applications for mixed reality on display at the Fair show great promise for enhancing efficiency and reducing cost in industrial applications, but a significant stumbling block was also evident. The convenience and power of the software was not matched by the display hardware. Many of the systems on display used headsets that were heavy, awkward, and uncomfortable to wear, making their use tolerable for short stints only. Other systems depended on tablets or smartphones as their display device, which, while more comfortable to deal with, left the user with only one hand free.

Ideally, mixed-reality display systems should be hands-free, comfortable to wear, totally portable, and would feature long-enough battery life to operate for at least a full work shift between charges. They should also provide high-quality virtual images in a way that avoids user eyestrain, include cameras and position sensors to provide context information such as where and at what the user is looking, and have high-bandwidth communications with the network providing the virtual images and object recognition AI. The displays should also be cost-effective to encourage their widespread adoption. Most of the display systems being demonstrated at the show fell short of meeting one or more of these requirements.

Vendors are actively working to resolve these issues, however, and promising display systems were at the Fair alongside the application software. Daqri, for instance, featured its own smart glasses design in conjunction with its Worksense AR software. The glasses have an integral headband and use a belt-clip computer and battery pack attached via an umbilical cord to handle communications and data processing. Lightweight (about 10 ounces), the glasses use transparent liquid-crystal-over-silicon (LCoS) displays in front of each eye to provide stereoscopic AR on top of the real-world image. The price is steep, though. The glasses, which come bundled with the software, run at $4,995 USD.

Vendors are working to overcome limitations in the current VR/AR goggle generation, aiming to increase battery life, improve comfort, and lower cost. (Source: Daqri)

Another display system on view at the show at software company exhibits came from Vuzix. The company’s M300 smart glasses, at a cost of $999 USD, are a bit more affordable but not capable of providing stereoscopic images. They have the shape of conventional (lens-less) eyeglass frames and present the AR image to either the right or left eye on a tiny screen. The design incorporates both its electronics and battery pack into the frames, although external USB batteries can be connected to expand the operating time beyond the one to two hours that the internal battery provides.

Still, the hardware available for industrial mixed-reality applications is far from ideal. But as the value of mixed reality gets proven by early adopters, the hardware challenges will see increasing resources being thrown at them. The hardware may be weak now, but the situation will undoubtedly improve as the ready and willing software proves its value among its early adopters.

Check out these other articles in the Industry 4.0 Special Report:
Hannover Fair shows off the factory technology of tomorrow — Four key trends emerged from the massive assemblage that is industry’s biggest annual showcase.

Robots are becoming more helpful, easier to use, and friendlier — At the Hannover Fair this year, industrial robots showed both utility and coworker compatibility.

3D printing gains industrial momentum — Once relegated to prototyping and hobbyist activity, 3D printing is evolving to address industrial requirements.

“Digital Twins” enable machine simulation & maintenance, Industry 4.0  — Marry EDA to the Industrial IoT to birth Digital Twins for system-level maintenance, simulation, and optimization.

Related articles from the ASPENCORE network:
Fear Not the Cobot — Discussing the newest and perhaps most exciting realm of industrial robotics: collaborative robots.
Meet the TI Robotics System Learning Kit  — An exciting new university robotics kit and teaching tools from TI will help prepare future engineers for system-level design.