BY DEBBIE SNIDERMAN, CEO
VI Ventures LLC
www.vivllc.com
Beyond consumer use in wireless selfie sticks, sports cameras, and event photography, drone imaging is finding its way into many commercial, government, and public safety applications. More drones/UAVs are being used to inspect high-voltage power lines replacing helicopters, an expensive and sometimes dangerous endeavor. They’re used in aerial cinematography, energy, forest management, mapping and surveying, oil and gas, and wildlife monitoring applications, as well as in covert operations, emergency response, payload delivery, security and surveillance, tactical operations, search and rescue, law enforcement, and even medical operations.
Aerial imaging has its own needs. Drone embedded systems require much more processing power than other mobile devices for handling flight control, advanced analytics, video stitching, and wireless networking applications. Systems-on-a-chip (SoCs) can record video while streaming live over Wi-Fi when previewing and sharing is needed on the ground. Software features like electronic image stabilization (EIS) and de-warp allow panoramic videos to be taken, reducing the need for mechanical gimbals. Many drone applications also require unique cameras, sensors, and optics.
What makes drone cameras special?
Ruggedized industrial drone cameras go into non-consumer UAVs. Brendan Lelieveld-Amiro, a sales engineer at industrial camera manufacturer Lumenera Corp., says that the specialized cameras differ widely from low-cost, consumer-level cameras in devices such as GoPro cameras, mobile phones, and DSLRs.
Many high-end drone applications need the cameras to provide images in raw format without lossy compression. Users running precision inspections need to trust that the data provided by the camera is accurate, whether for agriculture or inspection of hairline cracks in a water dam. Any processing to beautify, enhance, or compress an image could render the data useless for high-end commercial applications.
Industrial cameras also use highly precise and deterministic triggering. They trigger within microseconds from the time they are told to take the picture and when the exposure process begins. Lelieveld-Amiro says that the latency in consumer cameras in fully manual mode can range from 50 milliseconds to almost 1 second, which won’t cut it for industrial applications. Synchronization is important to tag the images with metadata from the flight system such as exact coordinates from the onboard GPS, velocity, altitude, and timestamp information.
Synchronization and timing are also crucial when drones carry multiple cameras for applications like precision imaging. All of the cameras pointing downward in a moving drone need to take the picture at the same time in order to overlap their images correctly.
Many precision-imaging applications use multi-spectral imaging systems. Separate cameras collect color images in the visible light spectrum and near infrared (NIR) spectrum while pointing in the same location. Image processing is used to stitch and align images to view different bands of response.
NIR is important in precision agriculture inspections in which, depending on the health of crops, NIR will be absorbed or reflected differently. Images allow users to create the normalized difference vegetation index (NDVI), a graphical indicator of crop health. NIR imaging is also used in drones that provide building inspections to reveal additional information of interest beyond the visible spectrum.
Drone cameras rated for commercial use are also ruggedized, able to withstand high levels of vibration, different ranges of temperature, and humidity, with higher levels of uptime and reliability than those not rated for commercial use, like digital SLR or point-and-shoot cameras.
Commercial industrial cameras don’t have onboard storage. They do contain memory buffers but transfer image data to an SoC or single-board computer to store and optionally process while flying. Based on image results, a drone can change its flight pattern if the software found a missed location or a poor image due to cloud coverage. They can even be used to ensure collision avoidance against power lines or other drones.
Filters are also available and used on industrial cameras, depending on the application. Mono cameras with clear glass are common, as well as color cameras, and customers can specify filters covering the spectrum from 380 to 1,000 nm. Some use visible blocking filters to look only at NIR information, and others use separate cameras for red, green, and blue and block the NIR. There are also discrete cameras for different wavelengths outside of this spectrum that can be used in parallel.
Monochrome camera sensors are still very important and are used in inspection applications. Using color cameras reduces the resolution of the image due to built-in Bayer filters and conversion or debayer/demosaicing algorithms that digitally process and reconstruct the full-color image. Mono cameras provide perfect pixel-to-pixel information without using the Bayer pattern or losing resolution.
Many UAVs are becoming piloted autonomously, so preventing or minimizing downtime is important. Cameras are certified to handle greater shock and vibrations than consumer cameras do and achieve a much longer mean time between failures.
While cameras in mobile phones include a lens in the same package, on commercial drone cameras, they are specified separately, attached, and controlled remotely. Instead of moving mechanical lens shutters that can wear out over time, they use electronic digital shutters. Software adjusts focus and iris for precise control and uniformity of all images taken during flight while outdoor conditions change.
The trend for consumer camera resolution is changing. It has historically been toward higher resolution, but, Lelieveld-Amiro explains, many mobile-phone manufacturers have stopped pushing that. “After a certain amount, it no longer makes sense. It’s too high most of the time, and the lenses included aren’t good enough to make use of the resolution,” he says.
In UAVs, the required resolution depends on the ground resolution. The higher it files, the higher resolution is needed to obtain, say, 1 cm/pixel at the ground. In certain commercial applications, regulations dictate how high a drone can fly in some states. Therefore, resolution and frame rates are important.
UAV camera resolutions range from 1 to about 29 megapixels, with the highest used in applications that fly high in the air or require very high ground sample distance (GSD). Precision agriculture, power line, and oil pipeline inspection cameras typically use cameras ranging from 2 to 12 megapixels.
For those that fly lower, there’s a trade-off between frame rate and resolution, and the choice depends on many things, such as the angle at which the camera is pointed, the oversample amount, and what the users do with the image data. Lelieveld-Amiro says that, at Lumenera, the highest-resolution cameras offer 6 fps and the lowest can offer 270 fps.
Lumenera Lt Series USB Camera.
Aerial cinematography: video on a drone
Stephan Ukas-Bradley, the director of technical marketing and Strategic Business Development at ARRI, adds that, in the digital mobile camera market for aerial coverage for motion pictures, dynamic range also becomes important. Users of the Alexa Mini camera need large pixels that are light-sensitive in order to emulate film. They don’t want to lose any details when shooting in the clouds and want to capture the most light even when the drone is pointing toward the sun.
“Drones are used when it’s impossible to get a shot with a helicopter because they’re smaller and can go in urban environments. Handheld mounting rigs and brushless gimbals keep the camera stable and allow very creative new things to be done, like filming from a second story in a building, then grabbing the camera on the street, then handing it over to a guy on a skateboard,” Ukas-Bradley says.
To protect against the outdoor environment, high-end drone cameras are subject to harsh testing far beyond sensors in small mobile devices. For example, Alexa cameras are hermetically sealed to prevent mist or sand from entering or causing damage and are exposed to vibration testing on industrial shaker tables with vibrations comparable to those of a truck running in rough idle. These cameras are also tested in environmental chambers from –20°C to 45°C and are designed to be used everywhere from the South Pole to the deserts.
The ALEXA Mini camera on a Freefly Alta drone was used for a short film, ‘The Journey,’ directed by Casey Warren and Danielle Krieger.
Airborne radiometry with thermal imaging sensors
Thermal camera imaging sensors are able to see minute temperature differences. On drones, commercial thermal inspections span a broad range of users. At the high end, looking at dams, pipelines, and power plants or performing search-and-rescue or firefighting inspections requires performance and safety. Lower-end users inspect roofing, solar panels, or real estate and may store their drone in the car.
Integration of FLIR's Duo Camera on a drone.
Jeff Frank, Senior Vice President at FLIR Systems, says, “While the sensor technology itself that we incorporate into our drone cameras is the same that we use in our handheld systems, the methods needed to get accurate airborne radiometric measurements are unique in several ways.” Sensors on drones often operate 50 to 100 meters away from the target, further than the 5 to 10 meters typical in handheld applications.
FLIR's Duo Camera integrated on the Phantom.
They can also be exposed to significant temperature variations as they fly at high speeds. Because variations in atmosphere affect transmission of thermal energy, these longer distances require additional processing to achieve higher levels of radiometric accuracy. Another concern for optimizing radiometric performance in drone cameras is managing airflow across the camera during flight.
The FLIR Duo Camera.
On the ground, cameras pan and tilt. UAVs provide a different dynamic and perspective on collecting imagery. Moving in 3D space, depending on how fast it is going, higher accuracy bidirectional interfaces are needed and metadata is included in the image info. The data is stored in a format easy to downlink to a ground station or view in real time.
FLIR’s Lepton thermal sensor used on the Duo camera. FLIR’s Duo Lepton camera combines thermal and visible imaging technology simultaneously in one camera and integrates them with data recorders on drones.
“Compared to other applications, military or automotive under-the-hood, drone applications are the least harsh,” he says. “Since the cameras are mounted on stabilized gimbals, unless it crashes, under normal circumstances, the physical environment is very benign. There is concern about air flow because rapidly moving air can impact image quality or add to the variation or quality of radiometry. It is key to compensate for atmosphere. Algorithms, flying methodologies, in-camera, and post-processing take that into consideration and are different in the drone world.”
Learn more about Arri
Learn more about FLIR
Learn more about Lumenera
Advertisement
