Machine vision: the next frontier for CMOS image sensors

By Majeed Ahmad, contributing writer

The technology advances of CMOS image sensors are unstoppable, literally. After redefining mobile phone designs for more than a decade, they have reached the doorsteps of artificial intelligence (AI) applications such as machine vision, object detection and recognition, and deep learning.

The availability of global shutter image sensors providing clear, sharp images at 4K camera resolutions is opening doors to new uses cases including 3D modeling, agricultural drones, industrial automation, and robotics. Agricultural drones, for example, capture high-resolution color images of crops and perform field monitoring in real-time.

Fig. 1: The new CMOS image sensors from Teledyne e2v (left) and SmartSens are eyeing machine-vision applications in intelligent traffic systems, drones, and smart barcode readers. Images: Teledyne e2v (left) and SmartSens Technology (right).

The breakthroughs in CMOS image sensor technology are enabling new camera designs for highly demanding machine-vision applications. And that’s allowing developers to replace the expensive charge-coupled device (CCD)-based image sensors, which have traditionally been used in applications requiring ultra-high resolution.

According to a recent study from IC Insights, CMOS image sensors continue to grab market share from CCDs, increasing their market share to 89% in 2017 from 74% in 2012. The market research firm also projects that 45% of CMOS image sensor consumption will come from applications other than camera phones.

That includes next-generation camera designs serving new use cases like augmented reality and virtual reality. Then there are object detection and recognition applications in advanced driver assistance systems (ADAS) and autonomous vehicle designs that are integrating CMOS image sensors and machine-vision algorithms on the same chip. The IC Insights study also forecasts that CMOS image sensors will experience the fastest growth in automotive designs during the 2017 to 2022 period.

New CMOS image sensor designs are enabling advanced digital imaging solutions in consumer and industrial machine-vision applications. Major advances in CMOS image technology include new pixel architectures, while many of these devices are integrating more functions to facilitate AI applications such as facial recognition and gesture control.

Evolution of pixel architecture
A wide range of customizable pixel architectures is bringing new dimensions into next-generation camera designs. These new structures offer a smaller optical format and faster photoelectric conversion and signal-readout functions.

A new breed of CMOS image sensors also provides more detailed views of a larger object surface, a critical requirement in industrial automation applications such as automated optical inspection (AOI) of displays in mobile phones, tablets, laptops, and TVs.

Fig. 2: OmniVision’s CMOS image sensors aim to boost pixel performance within the same optical format. Image: OmniVision Technologies, Inc.

But what makes these image sensors ready for vision-system–based automated inspection? What are the key enablers? Here is a sneak peek of the key features driving the transformation of CMOS image sensors and how they enable these devices to serve new use cases in the machine-vision realm.

1. Resolution. The high-resolution and high-performance global shutter image sensors eliminate motion artifacts and blurring, thus facilitating sharper images while supporting 4K and 8K camera designs. The global shutter function enables simultaneous image capture by all pixels to capture moving objects instantaneously without distortion. In other words, it’s capable of imaging all pixels at the same time. That, in turn, facilitates multi-viewpoint cameras with higher resolution.

These high-definition imaging solutions also feature precise color and uniform illumination over a larger area. The image sensors like ON Semiconductor’s X-Class also allow a single camera design to support multiple resolutions with different pixel functionalities. Therefore, larger pixels can trade resolution at a given optical format for greater sensitivity or lower noise operation.

2. Speed. Speed is critical in capturing moving objects, so the new CMOS image sensor designs offer a combination of high frame rate and resolution. These devices are designed for programmable high-frame-rate speeds that better respond to information retrieval demands in image-recognition applications like ADAS, factory robots, and industrial drones.

The new image sensors also feature high-bandwidth readout capability to bring greater design flexibility to machine-vision applications with real-time performance. The above- mentioned X-Class family of CMOS image sensors support two speed grades, one that utilizes 10-GigE interfaces to provide resolution speeds of up to 90 frames per second (fps) and the other that supports 27-fps speed while using USB 3.0 interfaces.

3. Sensitivity. The new pixel structure in Panasonic’s organic photoconductive film (OPF) image sensor offers both high-sensitivity mode and high-saturation mode, allowing camera systems to switch these modes according to the available light intensity. The high-saturation mode can capture as much as 10 times the high-sensitivity mode, but high-sensitivity mode offers extremely high-sensitivity pixels to improve imaging quality in low- and bright-illumination environments.

For instance, in low-light conditions, signals from adjacent pixels can be added to raise the sensitivity level. Likewise, while shooting bright scenes like daytime outdoors, the built-in signal-processing functions can perform array conversion to facilitate additional pixels in real time.

4. Low-noise designs. The low-noise pixel architectures boost shutter efficiency and allow global shutter operation to produce distortion-free images of fast-moving objects. The CMOS image sensors are now employing high-speed noise cancellation techniques to counter the noise created at the time of resetting the pixel. That allows them to quickly suppress this reset noise by using the negative feedback loop provided for each pixel.

Then there are image sensors like CMV50000 from ams, which feature on-chip noise-reduction circuitry like black-level clamping to efficiently capture high-quality images in low-light conditions. These low-noise sensors ensure smooth imaging operation without distortion in all capture conditions. Moreover, low-noise operations significantly boost the dynamic range and imaging sensitivity.

Convergence with machine-vision algorithms
The advent of machine learning is also taking the integration of image sensors with compute systems to a whole new level. The CMOS image sensors are bolstering the control and signal-processing functionality to enable real-time output and superior dynamic range. That allows image sensors to capture scenes in both bright and dark areas with minimal loss of detail.

Take the example of Newsight Imaging’s system-on-chip (SoC) design targeted at LiDAR and other machine-vision applications. It integrates a CMOS image sensor, machine-vision algorithms, and a pixel array on a single chip. The solution includes a DSP controller capable of performing complex calculations for depth-sensing and machine-vision tasks.

The NSI5000 chip from Newsight claims to significantly extend the dynamic range of the CMOS image sensor while retaining high accuracy, which allows it to bridge the gap between indirect time-of-flight (iTOF) known for shorter-distance and longer-distance automotive requirements. Newsight is also targeting these CMOS image sensor chips at drones and robots.

The computer-vision algorithms have been getting more sophisticated over the years, and so has the CMOS image sensor hardware. Their convergence will benefit new applications and use cases currently transitioning from CCD to CMOS imaging solutions. And it will help create more capable computer-vision designs at lower costs and smaller power consumption.

This clearly shows how the humble CMOS image sensor has come a long way from solely serving the camera in the phone. And that its evolving design framework is making it ready to serve the AI revolution with a lot more use cases and applications, including next-generation smartphones already employing CMOS image sensors in facial-recognition applications.