A new AI camera: 3D images are constructed by photon time difference, and single pixel imaging can reach 1000 frames per second.

Imagine such a scene, shouting at an animal, and then you can tell whether it is a dog or a horse from the echo. You may think that such a thing is far away from us, but a scientific team has completed this photographic effect.

In a new paper recently published in Optica magazine, researchers in Britain, Italy and the Netherlands describe a new way to make animated 3D images: by capturing the time information of photons, rather than their spatial coordinates.

The researchers extracted the three-dimensional image of the scene by adjusting the reflection time of light on the detector. This new technology called time imaging shows an important use of machine learning.

Time imaging system has some advantages over ordinary imaging. For example, the new system will take images very quickly and may work at the speed of 1000 frames per second; Moreover, this rough and fast 3D imaging may have many applications, such as being used as a camera for self-driving cars to improve the accuracy of pathfinding and emergency speed, and developing 360-degree perception for mobile devices and health monitors; Most importantly, this single-point detector for collecting time data is small, light and cheap.

Photos and videos are usually made by using digital sensors to capture photons (components of light), that is, ambient light will reflect an object, and the lens will focus it on a screen composed of tiny photosensitive elements or pixels. An image is a pattern of bright spots and dark spots produced by reflected light.

Take the most common digital camera as an example, it consists of hundreds of pixels, and an image is formed by detecting the intensity and color of light at each spatial point.

At the same time, several cameras are placed around the object to shoot the object from multiple angles, or the object is scanned by photon flow to carry out three-dimensional reconstruction, and a 3D image can be generated. But no matter which way is adopted, the image is constructed by collecting the spatial information of the scene.

In recent decades, researchers have invented a more ingenious method, which only uses a single pixel detector to capture images. In order to do this, they did not expose objects to uniform illumination, but to different illumination modes. These flashes are similar to the square bar code on the package.

Each pattern will reflect different parts of the object, so the light intensity measured by pixels will change with the change of the pattern. By tracking these changes, researchers can reconstruct images of objects.

Now, Alex Turpin, a data scientist at Glasgow University, Daniel Fajo, a physicist, and their colleagues have invented a method to generate a 3D image with a single pixel, but without a patterned flash. They use lightning fast single photon detectors to illuminate a scene with uniform flash and simply measure the reflection time.

The accuracy of the detector is 1/4 nanosecond, and it can calculate the function relationship between the number of photons arriving and time, so researchers can reconstruct the field only by using this information.

This is a novel method, because in principle there is no one-to-one correspondence between the arrangement of objects in the scene and the time information. For example, photons reflected from any surface 3 meters away from the detector will arrive within 10 nanosecond, regardless of any direction toward the surface.

The so-called time-of-flight camera is to increase the depth and make 3D images by accurately calculating the flash time of objects reflected to different pixels.

The new 3D imaging equipment starts with a simple and cheap single-point detector, which is adjusted as a stopwatch for photons. Unlike cameras that measure the spatial distribution of color and intensity, detectors only record the time required for photons generated by instantaneous laser pulses to bounce off each object in any given scene and reach the sensor. The farther the object is, the longer it takes for each reflected photon to reach the sensor.

The time information of each photon reflected in the scene (called time data by researchers) is collected in a very simple chart.

Then, with the help of complex neural network algorithm, these images are converted into 3D images. The researchers trained the algorithm and showed it thousands of routine photos of the team moving and carrying objects in the laboratory, as well as the time data captured by the single point detector at the same time. At the same time, they also took a real 3D image of the scene with a non-flying camera.

Finally, this neural network knows enough about the correspondence between time data and photos, so that a highly accurate image can be created only by time data. Compared with the time-of-flight camera, the time image is blurred and lacks details. However, it clearly reveals the human form.

Laura Waller, a computer scientist and electrical engineer at the University of California, Berkeley, said: "At first glance, this ambiguous approach seems to make the problem unsolvable. Single pixel imaging, when I first heard this concept, I thought, this should work. But on second thought, it should not work. "

Dr Alex Turpin, a data science researcher at the School of Computing Science, University of Glasgow, said: "If we only consider spatial information, and the single-point detector has no spatial information, then single-pixel imaging is impossible. However, this detector can still provide valuable time information. Unlike traditional image making, our method can completely separate light from process. "

Moreover, in order to achieve this goal, Alex Turpin and his colleagues adopted a machine learning program called neural network, which can image the moving people in the scene by itself after training the neural network with data sets.

Different from traditional cameras, single-point detectors used to collect time data are small in size, light in weight and cheap, which means that they can be easily added to existing systems, such as cameras of self-driving cars, to improve the accuracy of path-finding and braking response speed.

In addition, they can also enhance the existing sensors in mobile devices, such as Google Pixel 4, which already has a simple gesture recognition system based on radar technology, and can even use the next generation technology to monitor the ups and downs of hospital patients' chests, remind patients of breathing changes or track movements, and ensure their safety in a way consistent with data security.

Dr Alex Turpin added: "We are very excited about the potential of the system we developed, and we look forward to continuing to explore its potential. Our next goal is to develop an independent, portable and ready-to-use system. We are eager to start studying our options and conduct further research with the help of business partners. "

https://www . science mag . org/news/2020/08/time-camera-generates-3d-images-echoes-light

https://phys.org/news/2020-07-imaging-pictures.html

https://www.osapublishing.org/optica/abstract.cfm? uri = optica-7-8-900