Nice gesture!

BY ROLF WEBER, Senior Applications Engineer, Osram Opto Semiconductors, www.osram-os.com/

Most people are aware of sensors that recognize gestures or detect movements from their use in computer games. Manufacturers of laptops and tablets are working on similar technologies as a means of entering data and controlling these computers. The automotive industry is also looking to this new technology for its controls and safety systems. Depending on the application, solutions use intelligent reflex sensors, cameras or propagation time measurements to detect movement. This article looks at the technologies and components involved in each of the methods.

Kinect, X-Box, Wii — these are the names most often associated with the concept of gesture recognition. Three-dimensional recognition systems are in demand not only for gaming or for controlling computers, however. In industry, for example, 3D cameras are used to measure the precise size of packages on a conveyor belt. These systems are also used at automatic doors to detect whether a person is approaching the door or simply walking past. Vehicle manufacturers are working round the clock on methods for gesture control. They see enormous potential for intuitive control of many of the functions in the cockpit, with as little distraction for drivers as possible. In addition, new automobile safety systems, such as active pedestrian protection, are based on detecting movement. Apart from very few exceptions, all gesture and movement detection systems are based on optical solutions that use either intelligent near-field sensors or propagation time measurements and cameras to register objects.

Fig. 1: Automobile manufacturers are looking to use gesture recognition for more intuitive control of a wide variety of functions in the vehicle. 

Sensors with small ranges
Gestures for controlling tablet computers and notebooks are typically recognized at a distance of ten to a maximum of 30 centimeters. In the automotive sector, developers are also intent on finding solutions that operate within a narrowly defined range. This will prevent drivers or passengers inadvertently switching functions on or off. Simple gestures such as raining or lowering your hand to adjust the air conditioning should distract drivers as little as possible. The first solutions coming onto the market consciously make use of movements designed to give drivers time to become accustomed to this new control option. In the Golf VII, which was launchedlin the  fall of 2012, the central display detects whenever a hand approaches it. It then automatically displays the most important controls so the driver can immediately tap the appropriate function. There are plans to add simple swiping movements in the next generation.

Gesture control systems for these small distances can be based on camera sensors but are often, as in the case of the latest Golf, designed around optical reflex sensors. These sensors use the light reflected from a hand to detect movements such as approaching, receding, swiping, scrolling and diagonal motions. The sensors consist of several infrared LEDs (IREDs) which emit invisible light, a detector which registers the reflected light, and an IC which converts the detector signals into movement information. Compared with camera solutions, reflex sensors require far less computing power and therefore less current. This is an important aspect particularly for mobile devices such as tablet computers. Another benefit is the comparatively small space that these sensors take up.

A good example of such a solution for gesture recognition is the SFH 7770 from Osram Opto Semiconductors. This digital proximity and ambient light sensor combines two functions in one component and is ideal for use in displays.  The sensor detects approaching objects and their movement, and enables the display backlighting to be automatically adjusted depending on the lighting conditions at the time. At the heart of the SFH 7770 is an ASIC with two photodiodes that register the proximity signal and the ambient brightness. The sensor has three integrated LED drivers and can be operated with three successively pulsed IREDs to detect hand movements such as swiping or scrolling (Fig. 2 ). From each IRED it receives a reflected light pulse and uses the signal relationships to determine the movement information. The distance is calculated from the measured intensity of the reflected light. IREDs with a wavelength of 850 nanometers (nm) are suitable emitters because light at this wavelength is barely perceptable  to the human eye. The light has a weak red glow but, especially in short pulses, it is virtually invisible. The beam characteristics of the emitters depend on the application. Narrow-angle emitters are appropriate if the active zone is restricted to a narrow solid angle or if strong light signals are needed because the objects in question have low reflectance or because a relatively large range is needed. Broad beam angles are better for covering a wider area. In both cases, the optical output must be high enough to ensure that the signal at the detector is strong enough for reliable evaluation. A suitable IRED for this sensor is the infrared MiniMIDLED which bundles the light beam with an integrated reflector. It is less than one millimeter high and produces a radiant intensity of 60 milliwatts per solid angle (mW/sr) from a current of 100 milliamps.

Fig. 2: The SFH 7770 digital proximity and ambient light sensor can be operated with three infrared LEDs that can be flexibly placed so that it can detect not only an approaching object but also its movement. The direction of movement results from the signal relationships of the beams emitted from the individual LEDs and reflected from the object. 

Sensors with medium ranges
Working distances of up to a few meters are typical of computer games in which players control their avatars by means of body movements (Fig. 4). These systems register complex three-dimensional movements. This principle has now migrated to various other applications. The first televisions operated by gestures have now appeared on the market, and there are systems that allow surgeons to scroll through x-ray images on a monitor in the sterile environment of an operating theater without having to touch any of the controls. In vehicles, 3D sensors could be used, for example, to monitor the position and movements of the front-seat passenger so that the trigger strength of the airbag can be adjusted in the event of a collision. The classic way to determine the distance of an object is by triangulation, in other words, by illuminating or recording a scene from two different angles. This method is compute-intensive and is used only in special cases. Nowadays, digital camera sensors and appropriate software provide much simpler methods for 3D detection.

One of the most famous examples is the Kinect sensor. This is backed by technology from the Israeli company PrimeSense. It is based on an infrared laser that superimposes an invisible dot pattern on the scene in front of it and on a camera which records the image with the aid of a special detector chip. When an object moves in the field of vision of the camera, the dot pattern changes. The detector uses a complex algorithm to register these changes and determine the relative distance from the sensor. For the detector to operate irrespective of the ambient light conditions, a spectral filter allows only the laser light to reach the detector chip.

Another way to register distances is to use TOF (Time Of Flight) or PMD (Photonic Mixing Device) cameras. The cameras are operated in conjunction with an infrared light source. For each pixel, they measure the propagation time of the light from the light source to the object and back to the detector. The result is an image with distance information for each pixel. Because of the high speed of light (which covers 300 meters in one millionth of a second), it is generally not the actual propagation time of a light pulse that is measured. Instead, the infrared transmitter is modulated in continuous operation with around 20 to 80 megahertz (MHz) and the phase shift of the beam is registered. The accuracy of this distance measurement and the range of the sensor increase with the modulation frequency of the light source. Such cameras operate for example with high-intensity infrared LEDs such as those in the Oslon family which are based on high-power thin-film chips. These components are also qualified for use in the automotive sector. Thanks to nanostack chip technology, which produces almost twice as much light per chip, the Oslon is currently the smallest IRED with more than one watt of optical power.

Fig. 3: The Olson Black Series is currently the smallest infrared LED with more than one watt of optical power. 

Infrared laser diodes are also suitable as light sources for these applications. Pulse laser diodes have particularly fast switching times and can therefore be modulated with up to 80 MHz (approx. 30 MHz with IREDs). They therefore offer excellent measurement accuracy. The SPL TD 85-C with a wavelength of 845 nm and an optical output of almost half a watt is a good example of a laser diode that has been specially developed for 3D cameras. With its metal enclosure, this component complies with industrial quality standards.

Fig. 4: The SPL TD 85-C infrared pulse laser diode has been designed especially for 3D cameras and is qualified for industrial applications.

The sensor from the US company LeapMotion is based on another principle that has so far not entirely entered the public domain. It uses high-power IREDs with a wavelength of 850 nm and several cameras to register the position of all ten fingers with an accuracy, according to the manufacturer, of 0.01 mm. It can therefore recognize writing motions in the air or the tapping of a control.

Long-range applications: predictive driver assistance systems

Automobile manufacturers are using 3D sensors with ranges of ten and more meters for active safety systems. These sensors monitor the surroundings of the vehicle and can detect when a collision is about to happen. The driver is given a warning and emergency braking may even be applied. Initial solutions are already on the road, such as the Volvo City Safety system, which uses multiple laser beams to scan the area in front of the vehicle for objects at certain distances. In future, camera images will also be evaluated so these “pre-crash sensors” will be able to recognize the type of object involved, in other words whether the vehicle is about to crash into, say, a garbage can or a pedestrian, and will be able to determine the direction in which the object is moving. These developments are also being driven by Euro NCAP, the European New Car Assessment Programme. From 2016 on, only vehicles with predictive safety systems will receive full marks according to Euro NCAP's criteria. Cameras are virtually essential here.

Fig. 5: 3D cameras and laser scanners will be used to monitor the surroundings of a vehicle to predict collisions.

Some of these systems are based on the intelligent evaluation of conventional camera images. They often use additional illumination with infrared light to achieve good image quality in poor lighting conditions and therefore reliable evaluation. High-power IREDs with a wavelength of 850 nm are sources light sources for this purpose. We see light at this wavelength as a weak red glow so at these high optical outputs IREDs with a wavelength of 940 nm are used, particularly in systems which are operated with continuous light. This light is visible to the human eye only in absolute darkness but it is also less well registered by camera sensors. Other solutions use cameras to record the scene and determine the distance of objects using propagation time measurements. They are based on infrared pulse laser diodes or high-power IREDs with rapid switching times.

Summary
Gesture detection systems and 3D motion sensors are of interest for a wide variety of applications. Many of these solutions are based on illumination with invisible infrared light. These solutions would not be possible without the development of high-power IREDs and infrared lasers for lighting applications. The automotive industry in particular is making extensive use of this new technology. However, the automotive sector imposes the most stringent quality criteria, which means that many specific solutions have to be produced instead of relying on existing sensors.