Delivering high-quality camera phone images with dynamic range technology

Delivering high-quality camera phone images with dynamic range technology

Bringing the benefits of wide dynamic range technology to the CMOS image sensors used in mobile phones

BY YOSHITAKA EGAWA
Toshiba Semiconductor
and JOHN LIN, Toshiba America Electronic Components
Irvine, CA
http://toshiba.com/taec

Consumers are demanding more from their mobile phone cameras, which has led to explosive growth in the integration of high-pixel-count, high-quality cameras into mobile phones. To date, image sensors for cell phones have focused on delivering images taken under low-light conditions. However, with many phones now incorporating flashes to provide more light and shrinking pixel sizes enabling higher pixel counts in the same physical space, there is increased demand among mobile phone OEMs for high-performance, high-image quality sensors that can operate under bright light conditions. This demand is being met by the industry through the availability of wide dynamic range technology for low-cost CMOS image sensors. This article will detail and explain wide dynamic range technology and how a proper implementation will benefit CMOS image sensors used in mobile phones.

Dynamic range

The dynamic range of a sensor quantifies its ability to adequately capture both bright and dark scenes in the same field. It is defined as the ratio of the largest non-saturating input light signal (Imax) to the smallest detectable input light signal (Imin). It is often expressed in following logarithm scale:

Dynamic Range (DR)= 20* {log 10(Imax/Imin)}

(where I is the signal amplitude)

The dynamic range of solid-state image sensors varies over a wide range:

High-end CCDs > 78dB

Consumer-grade CCDs 66dB

Consumer-grade CMOS imagers 54dB1

Tonal range

The tonal range of a digital image is the number of tones it has in the output image. For example, in an 8-bit output, the potential range is 0 to 255. The actual range is determined by using a tone-mapping curve. Table 1 illustrates different tonal ranges and levels.

Table 1. Tonal range vs level

When a tonal curve is applied to the linear sensor data, the dynamic range and tonal range of the image can vary independently, depending on what tonal curve is applied. The tonal curve can compress the dynamic range, the tonal range, or both.

Benefits of wide DR

Dynamic range is directly proportional to the full “electron well” capacity of the pixel. More photons are collected for brighter areas. When the pixel wells are full, they overflow and detail is lost. This phenomenon is called blooming and causes “clipped highlights.” On the other hand, if the sensor reduces the exposure time to prevent further highlight clipping, many pixels that correspond to the darker areas of the scene might not have had enough time to capture any photons and might still have a zero value (hence the term “clipped shadows” is used). A wide dynamic range sensor can avoid such clipping problems and capture a much higher level of detail than a normal sensor. This is the main benefit of a sensor that features a wide dynamic range.

Solution

Wide dynamic range technologies have been developed for a variety of image sensor applications, but to meet the specific cost/performance needs of the mobile phone market, the technology must be implemented in a particular way. For example, wide dynamic range image processors for cell phones must be available at low cost in mass production quantities. Furthermore, they must support small pixel size and deliver a low noise, linear signal, and must not require frame memory.

A number of companies have developed wide dynamic range technologies to support the growing number of applications that can benefit from using the technology. Toshiba has developed an implementation of wide dynamic range technology specifically designed for mobile phones. More specifically, Toshiba’s technology, Dynastron-WD™, has four stages of operation to achieve the desired dynamic range.

1. Dual exposure operation on photodiode

2. Separate readout operation

3. Linear conversion synthesis operation

4. High signal compression

Dual exposure operation on photodiode

Figure 1 shows the pixel schematics behind wide dynamic range technology. This technology was first implemented in a 2.2 µm pixel cell structure.

Fig. 1. Shown above is the pixel schematics behind wide dynamic range technology.

Figure 2 shows the control timing and the change of photodiode well potential within one horizontal exposure period, i.e., one line time. ΦT (the exposure time) controls the photodiode (PD) potential level and timing. ΦS and ΦR control the readout.

Fig. 2. Shown above is the control timing and the change of photodiode well potential within one horizontal exposure period, i.e., one line time.

At t0, the well is reset to zero by applying Vh to ΦT and the electrons are discarded by ΦR. After that, the PD integrates up to t2 with Q (Charges) accumulated in the well. By applying Vm (Vm figure 2 .

Now there are two integration results QH and QL for TH and TL period. This is called dual exposure operation. By using dual mode operation, the well capacity is enlarged virtually by discarding some charges at t3 and recovering the signal later in the process.

Separate readout operation

The charge signal QH and QL can be separated by applying a separate readout mechanism as shown in figure 3. After the TH integration, the same control signal Vm is applied to ΦT at t6.

Fig. 3. The charge signal QH and QL can be separated by applying a separate readout mechanism.

The charge QH is then transferred to FD for output. After QH is taken by the ADC, Vh is applied to ΦT, QL is transferred to FD at t8 for ADC.

The two readouts happen within one horizontal line time (see figure 4). The signals QL and QH are converted by ADC separately and restored in separate line memory. In order to keep the same frame rate as conventional read-out, high-speed ADCs are required.

Fig. 4. T In order to keep the same frame rate as conventional read-out, high-speed ADCs are required.

Linear conversion synthesis operation

The next step in the process is to look at how to achieve the expected result from the two separate readout signals. At t3, by comparing the integration result to well potential level of Vm, there are three scenarios to consider as shown in Figure 5 .

Fig. 5. At t3, by comparing the integration result to well potential level of Vm, there are three scenarios to consider.

Under a high-illumination condition (a), which is the problem wide dynamic range solutions are designed to resolve, if there is enough well capacity, the integrated signal will go to point SF at t8. Due to the limitation of the pixel size, the signal will be clipped at somewhere below SF. By applying first Vm to FT, some of the charge is discarded. Within TH, signal SH is integrated. Assuming the illumination keeps no change for the entire horizontal period for most of the cases, the slope of the integration keeps the same, namely:

SH/TH = SF/TL

Then, SF=SH*TL/TH=B*TL, where only SH (TH integration result) and TL/TH are relevant.

Under a medium illumination condition (b), the well capacity is enough if the potential level is lower than that of Vm at t3, there is no charge discarded. The SF=SL+SH.

Under a low illumination condition (c), the overall integration level for TL is lower than that of Vm. The SH = 0, and SF = SL+SH = SL.

Now it is possible to conclude that the SF = Max (SL+SH, SH*TL/TH) for all scenarios.

High signal compression

In order to work with most of an image pipeline of 10 or 12 bits (typical of cell phone applications), signal compression is required. It might be combined with the tone mapping process by applying proper tonal curve in the image pipeline. The processing can be done on an SoC sensor (SoC sensor generally refers to a sensor die that contains the image processing capabilities) with support for wide dynamic range or in a third party ISP chip (in case of a Bayer-type sensor chip, i.e. a sensor die without any image-processing capabilities).