What is the magic behind the XR technology that allows Andy Lau and Jay Chou to perform without any sense of disobedience?

The 2021 CCTV Spring Festival Gala, as the party that Chinese audiences and even Chinese people around the world pay most attention to every New Year's Eve, is now not only a feast of literature and art, but also a grand event of scientific and technological innovation and a symbol of China's innovative technology. Display platform.

As technology increasingly becomes an important highlight of the Spring Festival Gala, this year’s Spring Festival Gala is particularly worthy of attention.

Jay Chou and Andy Lau appeared on the scene through AR and cloud technology. Take Andy Lau who is not in another place and Wang Yibo and Guan Xiaotong who are on the scene together to perform the program "Bulls Up" as an example. There is no violation in time and space. Jay Chou's song "Mojito" uses XR technology: the stage is moved to the seaside city, allowing characters, props and virtual scenes to interact and combine together. Let us see Jay Chou leading the band and driving a pink speed car to enjoy the summer style with the audience. ?

So what is XR?

XR is the collective name for VR, AR, and MR. Virtual Reality, Augmented Reality and Mixed Reality are bringing a series of changes to our lifestyle. In the not-too-distant future, you can also sit in a virtual conference room and hold a cloud meeting with your virtual friends, take a multi-person interactive course in a virtual smart classroom, conduct a simulation experiment, and even enter the virtual world. With real design and manufacturing, the boundary between virtual and real is no longer so obvious. All this benefits from the application of XR (VR, AR, MR) technology. ?

At the Spring Festival Gala, cloud actors participated in the performance through videos recorded in advance and packaged with technology. This method is not a simple connection or insertion, but an integral part of the program that is closely linked to the live actors. Many people said that AR technology is so awesome that Andy Lau was also invited to the scene. How is this achieved?

To put it simply, there are four main steps:

1. The program director needs to compose the shot in advance and design the actors’ movements and movements in each frame.

2. A dedicated team communicated and allowed Andy Lau to record in advance in a large studio.

3. Hundreds of software engineers added a large number of image effects to Andy Lau's shooting content, matched it with Andy Lau's singing and movements, and added virtual content to achieve "AR".

4. Guan Xiaotong and Wang Yibo perform in real time. When the content produced in the third step is superimposed with them, it becomes "XR".

At present, the cost of AR technology is relatively low. For example, lidar has been added to smartphones such as Apple's to achieve AR effects. However, XR technology has relatively few applications, requires a lot of computing power, and has a relatively high overall cost.