Why is one camera enough for some cameras and two for others?

Dual-camera structure

There are fewer and fewer highlights of high-end smart phones, and Lei Tutu can no longer show the advantages of mobile phones. So the more domineering dual cameras appeared. The dual cameras mentioned here are not like the previous smart phones, which have one camera in front and one camera in back, but a pair of eyes with two rear cameras to simulate people's eyes, so as to achieve more photo-taking functions and better photo-taking effects.

generally speaking, the current camera interface of smart phones is MIPI (mobile industry processor interface). In the past, there were only two MIPI interfaces on mobile phone platforms, one for the front camera and the other for the rear camera. To be a dual camera, the platform is required to support at least three MIPI interfaces. In fact, in the previous high-end platform, in order to achieve higher pixels, dual ISP(Image? Signal? Processing, image signal processor), for example, in order to support a 16M camera, two 8M capable ISPs will be used. This kind of platform is likely to have only 2 MIPI, but this can't stop engineers from doing front single camera+rear double camera.

dual camera application

here comes the question, what can dual cameras do?

1. Dual cameras can measure distance and can be used for distance-related applications

It is easy for human eyes to locate the distance of an object, but when one of them is closed, the positioning ability will be greatly reduced. Dual cameras are applications that simulate human eyes. To put it simply, to measure the distance is to calculate the angles θ1 and θ2 between the photographed object and the left/right Camera through an algorithm, and add a fixed Y value (that is, the center distance between the two cameras), so it is very easy to calculate the Z value (that is, the distance between the object and the camera).

However, it is also easy to calculate. If the center distance between the two cameras is too small, the computable object distance will be very close. If you want to calculate a longer distance, you must widen the distance between the left and right cameras.

As shown in the above figure, because the distance of the object can be judged by the algorithm of dual cameras, it is easy to make some special effects through this feature, such as:

A, blurred background

One of the most outstanding features of SLR cameras is the large aperture. Because the dual cameras can measure the distances of different photographed objects, aim at the objects that need large aperture, and blur other objects with different distances, the effect of large aperture can be easily realized.

)

B, background replacement

Because the distance can be measured, the subject in the photographed object can be extracted, and the background can be changed, so it is easier than PS to carry out matting.

C, background special effects

Because you can measure the distance and separate the theme from the background, it is easy to do any processing on the background, so I won't describe it too much here.

D, measured distance

This diagram clearly identifies the distance of different objects, and this distance information is identified with different colors. When AP obtains the distance information of different objects, it can achieve the above functions.

2. Dual cameras can do optical zoom

Optical zoom mainly means that the left and right cameras use different FOV (viewing angle), so the two cameras have different views. When users need wide-angle photos, they use a left camera with a viewing angle of 85 degrees to get a wide-angle effect. When users need telephoto photos, they use the right camera with a viewing angle of 45 degrees to obtain telephoto effect.

in order to make the objects photographed by the left and right cameras overlap highly, the dual-camera module of optical zoom cannot be too far away as the module of the camera used for distance application, but the left and right cameras should be placed as close as possible.

if the FOV of two Camera are different, one is large and the other is small, and then the effect between the two optical lenses can be achieved through an algorithm, optical zoom can be easily achieved.

If you don't use dual cameras, the text will be unclear after enlarging the picture

If you use dual cameras, the text will still be clear after enlarging the picture

This picture is a picture with a wide angle and a telephoto, and the middle attitude photo is calculated through an algorithm, so that the details are not distorted

3. The dark light effect is enhanced

Generally speaking, the dark light enhancement is done in two cameras: one. RGBG is used to get the color of the object, while the black-and-white camera is used to get a better amount of light to judge the light intensity of the object being photographed. Then the two images are fused to obtain better dark light enhancement.

At present, there are two fusion methods:

Black-and-white pictures are the main body, and the color of each pixel obtained from color pictures is pasted on the black-and-white pictures, and the two pictures are fused.

take color pictures as the main body, compensate the light intensity of each image obtained from black and white pictures to the color photos, and fuse the two pictures.

As to which method is more suitable for integration, different people may have different opinions, so we won't discuss it here.

similarly, for dark light enhancement, in order to make the objects photographed by the left and right cameras overlap highly, this kind of dual-camera module is required to be as close as possible.

It should be noted that Huawei P9 actually chose this mode of module

Of course, some people in the industry also said that the effect of this algorithm is not obvious at present. Dark light compensation is really helpful to users, especially when shooting at night. However, some customers think that Sony and Samsung's Dual PD technology is very good, and they are more willing to use Dual PD camera for dark light compensation.

whether dual cameras or Dual PD have good dark light compensation effect, you can compare Huawei P9 and Samsung's Galaxy S7 edge, and you will have the answer.

this usually uses a color+black and white camera. The brightness intensity of the picture is obtained by a black-and-white camera to compensate the dark light of the picture.

4. 3D shooting and 3D modeling

The algorithms of 3D shooting and 3D modeling are actually similar to the distance application, except that it requires higher accuracy, and sometimes even infrared ranging is needed to make more accurate distance judgment.

when it comes to the algorithm of dual cameras, I have to mention ISP(Image? Signal? Processing image signal processor), the main function of ISP is to post-process the signal output by the front-end image sensor, and its main functions include linear correction, noise removal, bad point removal, interpolation, white balance, automatic exposure control, etc. Only by relying on ISP can the details of the scene be well restored under different optical conditions, and ISP technology largely determines the imaging quality of mobile phones.

in the era of function machines, ISPs are all built on cameras, and cameras with different pixels are matched with ISPs with different performances. With the increasing pixel of mobile phone camera, the requirements for ISP performance are getting higher and higher. If ISP is integrated into the camera Sensor, the camera module will be too large, and even the photo effect will be affected. Therefore, in the era of smart phones, ISPs are generally on the main chip SoC. In order to achieve better results, some brand customers even add an ISP at any cost to achieve better and more professional photo effects.

a good camera algorithm needs a good ISP. ISP and algorithm complement each other and are indispensable. And dual cameras require more ISP performance. First of all, in order to make the signals of the left and right cameras be processed at the same time, a single ISP can no longer meet the needs of dual cameras. This requires a dual ISP to achieve this function.

taking dark light enhancement as an example, the color/black and white images enter their respective ISP channels and calibration channels, and then the two images are matched (for example, the same part of the two images is extracted, and the part captured by only one camera is removed), and then the related images are processed by algorithms such as occlusion, detection and compensation. Finally, the two images are fused to enhance the color. Of course, in fact, the ISP does more things with the algorithm than what is written in this picture.

of course, there is also a small episode in it. After all, there are two ISPs, and the two ISPs have some problems of different processing speed and processing ability. In order to ensure that two ISPs can sample at the same time, it is necessary that the pictures taken by dual cameras are taken at the same time. One of the solutions is to let the Sensor have a synchronization signal pin. The synchronization signals of the two cameras are docked, and each time a picture is read, the picture is stamped with a time stamp. ISP ensures that the photos taken by the left and right cameras are taken at the same time through the time stamp, and finally they are fused.

It's different from ordinary 3D movie shooting. The two cameras on the mobile phone can't produce enough visual difference in the process of image shooting, because the distance between the two cameras is different from that of the human eye. And in order to make people get 3D visual effects more obviously. Therefore, algorithms are often needed for enhancement.

Because the distance can be measured, the subsequent dual cameras can not only realize 3D photography, but also carry out 3D modeling. At this time, I think the importance of dual cameras will be even more important.

Other effects are enhanced, such as HDR and resolution improvement. In fact, these functions can be realized by a single camera, but dual cameras can make the effect better, so I won't list them one by one.

Summary:

At present, these functions are the most common functions of dual-camera mobile phones. Background blur/replacement, dark light effect brings more photo effects to users; Optical zoom makes us feel the camera function of zoom function; But personally, the most exciting thing in the future is the 3D function.

VR is so hot this year, where does the VR material come from? Still have to rely on the optimization of the dual camera algorithm. If the algorithm of 3D photographing and modeling is mature, dual cameras will become more popular.

ecological chain of dual cameras

In the previous article, we talked about the application and principle of dual cameras. Now let's focus on the industrial chain of dual cameras.

1. Dual-camera algorithm supplier

Because the algorithm needs to cooperate with ISP, the algorithm and ISP complement each other. If you want to do the algorithm well, you must have a good ISP.

as the main platform supplier, Qualcomm/MediaTek has its own ISP, so it has also developed its own dual camera algorithm. As for its algorithm, it remains to be tested by the market.

as Sensor suppliers, Sony, Samsung and OV are also actively developing dual-camera algorithms, and have not seen mass-produced products for the time being. However, in the era of functional machines, there is no ISP. When these Camera sensor suppliers make 2M/5M, they all have to match their own ISPs, so they all have their own experience in developing ISPs. Therefore, the development of dual-camera algorithm is relatively experienced.

Apple acquired Linx last year, and it also has patents and algorithms for multi-cameras. Whether it will be used in its own dual-camera model depends on this year's iPhone7. Theoretically, Linx has enough algorithms, so Apple won't go to a big factory to buy IP license for dual cameras.

In addition to these platforms, sensor suppliers and smart phone brands will develop their own dual-camera algorithms. Let's list other suppliers one by one:

Huajing, established in Taiwan Province in p>1996, mainly develops independent ISP chips. Some high-end mobile phones, cameras and cars all use the case of their ISP. Dual-camera phones also have their ISP's. Distance application, optical zoom and dark light compensation have all made achievements.

an Israeli company. The advantages of the algorithm mainly lie in optical zoom and dark light compensation. In depth of field, there is also some research. From the media propaganda, hTC has adopted its algorithm.

arcsoft was founded in 1994, with its headquarters in the United States and technical centers in Shanghai, Hangzhou and South Kyoto. The strengths of ArcSoft are optical zoom and dark light compensation. The second optical zoom picture and the dark light supplementary picture in the last article come from Irisoft.

Shanghai xingxinwei is a company established in 211, which is mainly engaged in the research and development of image processors. At present, its products are mainly used in the vehicle market. As one of the few companies that develop independent ISPs, they are also developing dual-camera algorithms and ISPs. After the dual-camera market, X-Chip will be a dark horse with great potential.

dual cameras are just coming out, so each family has its own advantages and disadvantages in terms of algorithm. However, judging from the effect of dual-camera mobile phones that have been launched at present, the algorithms of each family need to be improved. After all kinds of algorithm capabilities go up, it is bound to make dual cameras become the standard of mobile phones.

2. Dual-Camera Sensor supplier

Because two cameras need to synchronize time stamps when taking pictures, the camera sensor needs to have a synchronization signal. At present, the sensor suppliers with this synchronization signal are Sony, Samsung, OV and Gekewei. Therefore, the dual-camera Sensor mainly uses these products.

3. Dual-camera module suppliers

At present, there are many module factories that can make dual cameras, including Guangbao, Shunyu, Xinli, Namuga, O-Film, Samsung Electromechanical and Qiu Ti. However, those who have mass production experience are mainly Guangbao, Shunyu and Xinli. Huawei's models mainly use modules of Guangbao and Shunyu. And O-Film and Samsung Electromechanical, relying on their strong factory capabilities, are now making great strides in the field of dual-camera modules. Namuga maintains a good communication relationship with various algorithm companies, and is a supplier of Samsung mobile phones, and is gradually making efforts in the field of dual camera modules.

however, different functions require different modules. Let's continue to take the four functions mentioned above as examples:

The application of distance, in general, is the large and small cameras, and the common camera specifications are 13M+2M and 13M+5M.

Only in these two ways can we better calculate the distance between the subject and the lens.

The dual-camera module of optical zoom is mainly that the two cameras need to have different FOVs, similar to the following figure:

Different FOVs have different focal points, and then the optical zoom function can be realized through an algorithm. ?

In general, dark light enhancement is shot by an RGB full-color camera and a MONO black-and-white camera, as shown below.

In terms of algorithm, the amount of light input is mainly read by the black-and-white camera on the right, so as to compensate the color of the RGB camera on the left.

3D shooting and modeling, which is similar to the distance-related module. It's just that 3D requires higher accuracy of distance. In this case, it is more necessary to put the distance between the two cameras a little farther, and some even use infrared-assisted positioning to achieve distance measurement, and finally realize the role of 3D shooting and modeling.

Difficulties in making dual-camera module:

There are generally two ways to make dual-camera module: * * * substrate or * * * bracket. As shown in the following figure:

If the * * * substrate is used, two Camera Sensor*** * * are placed on the same substrate, and then an FPC is led out from this substrate. If * * * brackets are used, as shown above, the Sensors are fixed by brackets, and each sensor has its own substrate and its own FPC.

*** Advantages of the substrate: two S.