Augmented and virtual reality are high on the avionics technology checklist

By Majeed Ahmad, contributing writer

The coronavirus pandemic hit the airline industry at a time when it was in the middle of an ambitious pursuit of adopting new technologies, including artificial intelligence (AI), machine learning (ML), augmented reality (AR), and virtual reality (VR). The avionics industry, which has been slow to adopt multi-core processors due to certification complexities, had finally started to embrace powerful processing chips to allow designers to do more with less and run multiple applications on a single hardware platform.

Now, while the airline industry won’t be the same after the pandemic, it’ll be worth seeing if the convergence of compute power, sleek displays, and smart sensors makes avionics nimbler and more responsive to the emerging challenges in a time of social distancing and contact tracing.

Let’s begin with AI and its offshoots like machine learning and see how these technologies can help streamline the visual clutter in the cockpit and bolster the efficiency of staff tasked with aircraft maintenance and overhauling jobs.

Machine learning in avionics
Avionics, one of the industry leaders in adopting automation initiatives, is leveraging AI technology to move toward greater autonomy. The new efforts are aimed at reducing the pilot’s workload, accurately locating potential issues, and providing pilots with better situational awareness.

Collins Aerospace has joined hands with embedded software supplier Wind River to employ AI and ML technologies in aircraft cockpits to bolster intelligence of automated systems (Fig. 1 ). The two companies are also collaborating on how AI-based solutions can address the safety challenges related to determinism and predictability.

Fig. 1: The AI-centric machine-vision features provide pilots with a unique out-of-the-window view that is crucial in difficult landing conditions. (Image: Collins Aerospace)

The adoption of AI technologies like machine learning is now reaching the cockpit to process complex information from multiple sensors and sources and then presenting it to the pilot in a way that enables him or her to make intelligent decisions. Such systems can be segmented into two major categories: real-time systems and non-real-time systems.

Non-real-time systems can turn the aircraft into an e-aircraft by transmitting the in-flight data to an airline data center via satellite communication links. Likewise, 4G and 5G network links can be used while the airplane is on the ground at the airport terminal gate. The cloud-based remote analysis can monitor sensor data to ensure that the aircraft is operating efficiently and can recommend predictive maintenance if and when necessary.

On the other hand, in real-time avionic systems, machine-vision features can provide an intuitive 3D view of terrain maps to assist pilots, for instance, on a landing approach to airports. Such machine-vision systems, deployed on aircraft’s local processing platforms, can provide a deterministic, low-latency response to the pilot.

Augmented piloting
Alternative reality — which encompasses AR, VR, and mixed reality (MR) technologies — is becoming the next big computing platform in military and aeronautical designs. MR, which operates at the crossroads of AR and VR technologies, is about compounding real-world and digital interactions simultaneously.

AR, which digitally places virtual objects in the real world, is becoming a reliable commodity for surveillance and monitoring activities. For instance, pilots in the cockpit can view live airplane data through AR glasses. There are also tools that augment other aspects of decision making for pilots in real time.

For a start, AR adoption makes the complicated job of piloting a little easier. AR-assisted flying enables pilots to make safer and more informed decisions. That’s because AR enables hands-free movability and real-time information in ways that streamline the otherwise complex job of piloting airplanes.

The pilot, equipped with an AR head-mount display, can use the digital information that appears in the field of view to check through the pre-flight procedures visually. Moreover, in dark and rainy conditions, the vision systems overlay a crisp image of the surrounding areas to facilitate a smooth takeoff and landing.

Also take the case of helicopter pilots flying in low-visibility weather conditions. Here, AR brings the digital topographical information about mountains, houses, etc. in the pilot’s field of view while he or she is focusing on the path ahead.

It’s worth noting that AR glasses and headsets are also incorporating AI and 3D sensor technologies to scan, identify, and track problems that impact both airplane flight and maintenance operations in a variety of ways. For example, AR technology enables aerospace engineers and technicians to complement aircraft-maintenance routines by conducting smart visual inspections.

Virtual cockpit
AR’s more famous cousin, VR, in which the user is immersed in an entirely artificial world, is being widely used for flight training. Aviation training has seen minimal changes and optimization since the 1980s. That’s why Airbus and other avionics industry players are starting to explore VR technology for higher-value training.

The VR environment simulates the physical world very effectively, which allows airlines to embed practical training in a virtual sense and train pilots, engineers, and cabin crew more effectively. VR gadgets like head-up displays (HUDs) also enable ground staff to accurately see drop zones, runways, and other targets regardless of weather conditions.

Qantas and Lufthansa are among the first airlines to use VR for training pilots, engineers, and cabin crew. Here, aircraft engine manufacturers like Rolls-Royce are providing specialized VR training tools for serving a variety of avionic use cases.

Pilots are immersed in a virtual cockpit, and all they need is a personal computer, a VR headset, and a pair of hand controllers to conduct this training online (Fig. 2 ). The initial VR training makes pilots more productive when they move to the full-flight simulator and live-flight stages.

Fig. 2: A pilot is conducting VR-based flight training from home. (Image: Airbus)

Airbus is employing VR technology to complement simulator training. It enables trainee pilots to receive real-time feedback on flight simulations when they take a specific action. Furthermore, pilots can quickly learn engineering-related tasks through VR applications.

The future of avionics
AR and VR technologies, aided by AI and 3D scanning capabilities and features, are no longer the cool technologies creating a wow factor. They are finding practical use cases, and the avionics industry is providing new exploration venues. These technologies are helping in a range of applications from providing pilots with better situational awareness, surveillance, and monitoring to accurately classifying engineering and maintenance issues and flight training.

Aviation companies such as Airbus and Boeing have large and clean datasets that they can put to use to test and train AI algorithms with the help of multi-core processors with exceptional compute powers and AI accelerators targeted at specific machine-learning capabilities. That, combined with an array of AR and VR applications, could well transform avionics in the post-coronavirus era.