Photographic three-dimensional model

Not long ago, a group of researchers at the University of Washington released a new experimental technique. This technology can directly cut people from static 2D images and convert them into 3D animations, and finally introduce information into the real world in the form of AR. This fragrance technology is named Photo Wake-Up, which is suitable for photos, posters and even graffiti art. It can inject life into the embedded Pinocchio and make these graphics walk, run and even jump freely in the real environment.

In an interview with MIT Science and Technology Review on June 5438+February last year, the team said: "We think this method can not only provide people with a new way to appreciate and interact with photos, but also bring a new way to reconstruct virtual avatars based on a single image, and even understand the real state of human art modeling from a single photo."

In fact, the idea of reconstructing 3D models based on 2D images is not new. As early as more than ten years ago, the Stanford 3D reconstruction team led by Andrew Ng studied this topic. However, the emerging solutions are more powerful than ever (or the demand is more urgent), and they are closely related to the content supply problem of the current youth VR and AR industries.

Up to now, most 3D models for VR and AR content are still created by 3D artists by hand, which is a time-consuming, expensive and inextensible work. In addition, there is still a shortage of talents in the industry, and this problem cannot be alleviated with the expansion of scale. Because of this, all kinds of 3D buybacks represented by Sketchfab are widely welcomed, and artists can publish, share, discover and buy and sell all kinds of 3D, VR and AR content here.

Nevertheless, the creators are still eager to get rid of the dependence on human labor and hope to get more ideal benefits from those tools that provide automation capabilities. Specifically, these tools must be able to create shortcuts and "know-how", thus greatly speeding up and improving the creative process.

In this regard, Jan-Michael Trssler, founder and CEO of Trnio, said: "For a truly immersive world, the best content creators need to use a lot of technology and software to create real elements. I have seen many artists use photography to measure and capture objects, and then use it as a starting point for creating 3D models. Next, the artist will simplify and optimize the asset elements so that they can play a role in the AR/VR engine. "

Nowadays, the three-dimensional scanning technology represented by photogrammetry and volume capture is developing continuously in terms of output quality and shortening the production cycle of three-dimensional models, making the real world more and more easy to become the direct source of three-dimensional model making. This technology can help current creators, even any newcomers who are interested in it, or potential participants who are still hesitating, more easily complete the image outlined in their minds.

Charles Carriere, founder and president of Scancandy in New Orleans, pointed out: "We believe that most content creators don't actually explore 3D content because they lack such tools. In fact, almost all the content being created now exists in 2D form, because everyone can easily access and use 2D tools-the most important of which is the mobile phone camera. As long as we can provide platforms and tools for these creative Snapchat, Instagram and YouTube cows to create 3D models, high-quality 3D content will explode and attract more consumers. "

Based on the actual situation in the past, such expectations are obviously very unrealistic. However, the emergence of 3D reconstruction technologies such as photo awakening suddenly makes 2D images, assets and even resource libraries an inexhaustible source of materials. 3D models can be generated almost automatically and quickly applied to immersive experience scenes. In fact, this technology not only makes the whole industry catch up, but also is expected to trigger a Cambrian-like life explosion in the immersive field.

Focusing on the present, AR retail and shopping will become the first important beachhead position. A team of senior entrepreneurs, composed of Apple, Facebook, PayPal alumni and several Stanford University PhDs, recently made a collective appearance as a participant of Threedy.ai. Threedy.ai is a deep technology start-up, which specializes in solving the problem of 3D content supply.

Nima Sarshar, co-founder and CEO of Threedy.ai, explained, "You can think that the manufacturers who created these products must have corresponding 3D model materials, but the facts tell us that although there are relevant CAD files, relevant manufacturers are willing to make them public and generate 3D grids, but they lack corresponding textures or materials. In addition, there is no unified workflow for AR model creation, which can digest the specific tools of six major CAD suppliers and the file formats exceeding 14. The whole manufacturing industry is dealing with hand-textured packaging in a large number of outsourcing ways. "

For example, the Houzz AR application allows users to view furniture in a room based on indicators such as size and color. The challenge is that all advanced AR shopping applications, including Houzz, Wayfair and Overstock, only have a small number of 3D models in stock. Specifically, Houzz's 3D model only accounts for 3% of all restaurant furniture categories.

Sarshar explained, "Our goal is to become a Getty Images website in the field of 3D models of commercial products. You can imagine that in the future, every product in the Amazon catalog may have a corresponding 3D model, and how big this resource pool will be. "

Their first product is Threedy Convert, which can automatically convert ordinary 2D photos of household products and furniture into high-quality 3D models through special computational geometry and deep learning algorithms. This technology can be applied to the growing product categories in batches, usually only one product photo is needed, and the whole implementation cost is nearly two orders of magnitude cheaper than the existing solutions.

Sarshar pointed out: "Although scanning can provide higher quality, the whole process is still very boring and expensive. In addition, the conversion from high polygon scanning results to low polygon model suitable for XR is by no means easy. There is another important question. For most e-commerce websites, there is no corresponding physical object at all, only a limited number of photos of unstructured products. "

This method of starting with low-dimensional materials and helping to build high-dimensional materials is not only surprising, but also will be enthusiastically sought after by the market. This technology is not a short-term feasible remedy, but represents a brand-new content conversion channel. With the continuous development and changes of the VR and AR industries, these contents are expected to fully push the rendering experience to the general public at the right time-I am optimistic that all this may happen this year.