CN111510701A - Display method, apparatus, electronic device, and computer-readable medium for virtual content - Google Patents

Abstract

The present application discloses a method, device, electronic device and computer-readable medium for displaying virtual content, and relates to the field of display technology. The method includes: acquiring an image of a real environment acquired by an image acquisition device; The product description information corresponding to the object; obtain the visual presentation information of the target object in the real environment; generate virtual content according to the product description information and visual presentation information, and the virtual content is used to represent the product description information; add the target object at the position of the real environment virtual content. Therefore, in the real environment, the position of the target object passes through the added virtual content, so that the product description information corresponding to the target object can be displayed through the virtual content. Compared with the way of reading the product instruction manual, the way of the virtual content is more vivid. Interesting, and more able to attract users and facilitate users to learn the product description information.

Description

Display method, apparatus, electronic device, and computer-readable medium for virtual content

technical field

The present application relates to the field of display technology, and more particularly, to a method, apparatus, electronic device, and computer-readable medium for displaying virtual content.

Background technique

At present, users often read product-related introductions when they need to know about a product. For example, when installing a product or using a product, it is necessary to read the instruction manual of the product to successfully install or use the product. For another example, when shopping online, you need to watch the video introduction of the product or read the detailed introduction of the product. However, product-related introductions are relatively obscure to read, contain a lot of content, and are not easy to understand, and users are often impatient to read the content carefully.

SUMMARY OF THE INVENTION

The present application proposes a method, apparatus, electronic device, and computer-readable medium for displaying virtual content to improve the above-mentioned defects.

In a first aspect, an embodiment of the present application provides a method for displaying virtual content, including: acquiring an image of a real environment collected by an image acquisition device; if a target object exists in the image, acquiring product description information corresponding to the target object ; Obtain the visual presentation information of the target object in the real environment, and the visual presentation information includes the pose information of the target object relative to the image acquisition device; According to the product description information and the visual presentation information generating virtual content, where the virtual content is used to represent the product description information; adding the virtual content at the position of the target object in the real environment.

In a second aspect, an embodiment of the present application further provides an apparatus for displaying virtual content, where the apparatus includes: a first acquiring unit, a second acquiring unit, a determining unit, and a processing unit. The first acquisition unit is used to acquire the image of the real environment collected by the image acquisition device; the second acquisition unit is used to acquire the product description information corresponding to the target object and the location of the target object if there is a target object in the image. The visual presentation information in the real environment, the visual presentation information includes the pose information of the target object relative to the image acquisition device; the determining unit is configured to generate virtual content according to the product description information; the processing unit is configured with to add the virtual content at the location of the target object in the real environment.

In a third aspect, embodiments of the present application further provide an electronic device, including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more application programs configured to perform the above method.

In a fourth aspect, an embodiment of the present application further provides a computer-readable medium, where the readable storage medium stores program code executable by a processor, and when the program code is executed by the processor, causes the processor to Perform the above method.

The method, device, electronic device and computer-readable medium for displaying virtual content provided by the present application acquire an image of a real environment collected by an image acquisition device, determine whether there is a target object in the image, and if there is a target object in the image, the target can be acquired The product description information corresponding to the object, the product description information is the information used to describe the target object, and then the visual presentation information of the target object in the real environment is obtained, and the visual presentation information includes the relative position of the target object. pose information of the image acquisition device, generate virtual content according to the product description information, and add the virtual content at the position of the target object in the real environment. Therefore, in the real environment, the position of the target object passes through the added virtual content, so that the product description information corresponding to the target object can be displayed through the virtual content. Compared with the way of reading the text content related to the product, the way of the virtual content More lively and interesting, and more able to attract users and facilitate users to learn the product description information.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 shows a schematic diagram of an AR device provided by an embodiment of the present application;

FIG. 2 shows a schematic diagram of an AR device provided by another embodiment of the present application;

FIG. 3 shows a method flowchart of a method for displaying virtual content provided by an embodiment of the present application;

FIG. 4 shows a schematic diagram of virtual content provided by an embodiment of the present application;

5 shows a method flowchart of a method for displaying virtual content provided by another embodiment of the present application;

FIG. 6 shows a schematic diagram of virtual content provided by another embodiment of the present application;

FIG. 7 shows a schematic diagram of virtual content provided by another embodiment of the present application;

FIG. 8 shows a method flowchart of a method for displaying virtual content provided by another embodiment of the present application;

Fig. 9 shows the method flow chart of S840 in Fig. 8;

FIG. 10 shows a schematic diagram of an analysis model provided by an embodiment of the present application;

FIG. 11 shows a schematic diagram of the training and application process of the analysis model provided by the embodiment of the present application;

FIG. 12 shows a schematic diagram of virtual content provided by still another embodiment of the present application;

FIG. 13 shows a block diagram of a module of an apparatus for displaying virtual content provided by an embodiment of the present application;

FIG. 14 shows a block diagram of modules of an apparatus for displaying virtual content provided by another embodiment of the present application;

FIG. 15 shows a block diagram of a module of an electronic device proposed by an embodiment of the present application;

FIG. 16 shows a storage unit for storing or carrying a program code for implementing the method according to the embodiment of the present application according to the embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. At present, users often read product-related introductions when they need to know about a product. For example, when installing or using a product, you need to read the product's instruction manual to successfully install or use the product. For another example, when shopping online, you need to watch the video introduction of the product or read the detailed introduction of the product. However, the introduction related to using the product is relatively obscure to read, contains a lot of content and is not easy to understand, and users are often impatient to read the content carefully. For example, in real life, many commodities, such as household appliances, require reading instructions, which is often a tedious step.

In order to improve the convenience for users to read the product manual, reduce the tediousness of reading the manual, and more attract users to read the product manual and improve the convenience for users to learn the content of the product manual, augmented reality (AR, Augmented Reality) technology can be used to display. the contents of this product manual.

Augmented reality is a technology that increases the user's perception of the real world through information provided by a computer system. It superimposes computer-generated virtual content, scenes or system prompts and other content objects into the real scene to enhance or modify the real-world environment or representation. For the perception of data in the real world environment, users can observe the effect of augmented reality or mixed reality after the content such as virtual content is superimposed with the real world in the real world environment by wearing head-mounted display devices such as AR glasses.

For example, some AR glasses can collect information in the real environment through the cameras and sensors of the AR glasses. After the processor and specific algorithms are run, the corresponding images are rendered and displayed on the lenses, allowing users to feel the virtual image and the real world. coincide.

Some AR glasses have a split structure, that is, the processor module of the glasses is not located on the glasses, but is located on a separate computing unit, which is connected to the AR glasses through wires. The data collected by the cameras and sensors of the AR glasses are preprocessed and sent to the computing unit. The computing unit renders the corresponding virtual image according to the collected information, and the virtual image is transmitted to the AR glasses for display.

As shown in FIG. 1 , which shows an AR device, the AR device may be a head-mounted display device, and specifically, the head-mounted display device is AR glasses. As shown in FIG. 1 , the head-mounted display device includes a display screen 110 , a frame 120 and an imaging device 130 .

The frame 120 includes a front surface 121 , a side surface 122 and a rear surface 123 , the display screen 110 is mounted on the front surface of the frame, and the imaging device 130 can display the image of the virtual content on the display screen 110 . For example, the imaging device 130 may be a diffractive optical waveguide capable of projecting an image onto a display screen.

As an embodiment, the display screen 110 may be a lens of the AR glasses, and the display screen 110 may also transmit light, that is, the display screen 110 may be a transflective lens, and the user is wearing the head-mounted display device. When an image is displayed on the display screen 110 , the user can see the image displayed on the display screen 110 and can also see objects in the real world in the surrounding environment through the display screen 110 . Then, through the transflective lens, the user can superimpose the image displayed on the lens with the surrounding environment, thereby realizing the visual effect of augmented reality.

When the user wears the head-mounted display device, the display screen 110 is located in front of the user's eyes, that is, the front surface 121 is located in front of the user's eyes, the rear surface 123 is located behind the user's eyes, and the side surface 122 is located at the side of the user's eyes.

In addition, a front-facing camera is provided on the front surface 121, and the front-facing camera is used to perceive the environmental information ahead, so as to realize Simultaneous Localization and Mapping (SLAM), thereby realizing the visual effect of augmented reality or mixed reality.

In other AR technologies, front-facing cameras can be used to combine real scenes with virtual content. Specifically, the field of view direction of the front camera on the front surface of the head-mounted display device may be consistent with the direction of the user's field of view when the user wears the head-mounted display device, and the front camera is used to collect the image of the real scene, and the The collected image of the real scene is processed and displayed on the display screen in front of the user. Specifically, it can be an image of virtual content superimposed on the image of the real scene, and is seen by the user, so that the user can observe the vision of augmented reality. Effect.

As another implementation manner, the AR effect can also be achieved through a mobile terminal or a terminal with a screen, such as a tablet computer or a computer device. Specifically, as shown in FIG. 2, FIG. 2 shows another AR device. The AR device may be a user terminal, and the terminal includes a camera and a screen. As shown in FIG. The image of the scene (indoor scene, such as the lamp, sofa, etc. as shown in FIG. 2 ) can be the image collected by the camera of the terminal, and the display content corresponding to the virtual content A is displayed on the screen, and the display content can be a A picture can be added to the image corresponding to the real scene displayed on the screen, and the picture corresponds to the virtual content A, then the user can see through the content displayed on the screen that the virtual content is set in the real scene. Content A (such as a sphere in FIG. 2 ) to obtain an augmented reality display effect in which virtual content A is placed in a real scene.

Specifically, the steps of realizing the effect of augmented reality through the terminal shown in FIG. 2 include tracking, scene understanding and rendering. The implementation of the tracking may be to provide the relative position of the terminal in the real environment. Specifically, an accurate view of the location of the terminal and the orientation of the device can be provided with the help of visual inertial odometry, which uses camera images and device motion data.

Scenario understanding refers to determining properties or characteristics of the environment around a device. For example, surfaces or planes in the real environment can be determined through functions such as plane detection. such as a floor or table. In order to place virtual content, the terminal also needs to provide hit testing capabilities. This function can obtain the intersection point of the real-world topology, that is, obtain the specific position of each object in the real environment, such as the height of the table above the ground or the distance from the terminal, etc., so as to place virtual content in the real environment. Finally, light estimation can be done for scene understanding. Lighting estimation is used to correctly illuminate virtual content to match the real world, for example, when virtual content is placed in the real world, it will display the shadow of the virtual content under the lighting of the real world, thereby increasing the realism of the virtual content and matching with the real world.

The real environment displayed on the above-mentioned screen may be an image of the current environment collected by the camera of the terminal, or may be an image of a real scene sent by other terminals received by the terminal in real time. During video interaction, what is displayed on the screen in FIG. 2 is the real-time image of the real scene where the terminal is located, which is captured by cameras of other terminals in real time.

In order to achieve the above effect when a user reads a product specification, an embodiment of the present application provides a method for displaying virtual content, which can be applied to the devices shown in FIG. 1 and FIG. 2 . In this embodiment of the present application, the method may be applied to the electronic device shown in FIG. 2 , that is, the execution body of the method may be a processor or a client in the electronic device. In addition, it should be noted that the execution body of the method may also be the head-mounted display device shown in FIG. 1 . Specifically, the head-mounted display device is provided with a processor, and the execution body of the method may be the head-mounted display device. The processor inside the device. As an implementation manner, and in order to better illustrate the effect of this embodiment, for example, the execution subject of the method in this embodiment of the present application is a mobile terminal, such as the electronic device shown in FIG. 2 , however, this does not limit the The application field of the method of the present application does not limit the equipment and application environment to which the present application is applicable.

Please refer to FIG. 3. FIG. 3 shows a method for displaying virtual content provided by an embodiment of the present application, which is used to display virtual content at the position of a target object, so as to improve the convenience for users to read the product manual and reduce the time and cost of reading the manual. It is more tedious and more able to attract users to read the product manual and improve the convenience for users to know the content of the product manual. Specifically, the method includes: S301 to S304.

S301: Acquire an image of the real environment collected by the image collection device.

Wherein, the image acquisition device may be a device capable of acquiring images, such as a camera, a digital camera, or the like.

As an implementation manner, the image acquisition device can be installed on the above-mentioned head-mounted display device, or can be installed in a mobile phone. If the embodiment of the present application is applied to a head-mounted display device, the image acquisition device may be an imaging device installed on the head-mounted display device. Then the image acquisition device may be a camera installed on the user terminal.

As another implementation manner, the image acquisition device may be set independently, that is, it may not be provided on the execution body of the method. In this embodiment, the image acquisition device is connected to the execution body of the method of the present application, and is used to The collected image of the real environment is sent to the execution body of the method of the present application.

S302: If a target object exists in the image, obtain product description information corresponding to the target object.

The product description information may be product-related introduction information, and the information may include product-related information such as product usage instructions, installation instructions, feature introductions, and function introductions. Users can understand or use the product by reading the product description information.

As an implementation manner, the execution body of the method of the present application may determine that there is a target object in the image. Specifically, taking a user terminal as an example of the execution subject of the method of the present application, the user terminal obtains an image of the real environment collected by the image collection device, and determines whether there is a target object in the image collected by the image collection device. Specifically, the identity information of all objects in the image is analyzed to determine whether target identity information exists in the identity information of all objects, and if so, it is determined that the target object exists in the image. Then, the product description information corresponding to the preset identity information matching the identity information of the target object is searched in the data set, as the product description information corresponding to the target object. Wherein, the identity information can be the outline of the object image in the image, or the feature point of the object image in the image, or other information that can distinguish the image of the object from the images of other objects. Do limit.

The target identity information may be the identity information of the target object, and specifically, the target identity information may be preset. As an implementation manner, a data set is preset, and the data set includes multiple preset identity information and product description information corresponding to each preset identity. Wherein, the preset identity information is a preset product with product description information set, or the preset identity information is a preset product that needs to display product description information for the user.

As another implementation manner, the data processing server may also determine that the target object exists in the image. Specifically, the image collection device may send the collected image to a data processing server, and the data processing server determines whether there is a target object in the image according to the above method, and sends the result to the execution body of the method of the present application.

In addition, the product description information corresponding to the target object may include information such as installation guidance, operating instructions, and product introduction of the target object, which will not be repeated here.

As an implementation manner, an implementation manner of acquiring the product description information of the target object may be to send an acquisition request to a server corresponding to the target object. The target object may be a controlled device within the Internet of Things. The Internet of Things is a network concept that extends and extends its user end to any item and item on the basis of the "Internet concept" for information exchange and communication. With the development of IoT technology, some scenarios can be configured in the IoT system. For the configuration scenario, multiple controlled devices may be involved, and multiple controlled devices have a certain linkage relationship and can work together.

Among them, the controlled devices can be projectors, projection screens, smart lights, smart sockets, human sensors, door and window sensors, wireless switches, air conditioner companions, smoke alarms, smart curtain motors, and user terminals such as air purifiers and smart speakers. In an implementation manner, in the Internet of Things system, the controlling electronic device (such as the above-mentioned user terminal) may implement data interaction with the controlled device by directly establishing a wireless connection with the router. Furthermore, the electronic device can also realize data interaction with the controlled device through the data link between the cloud and the router after establishing a connection with the cloud. As another way, the controlled device can establish a wireless connection with the router through the gateway. The data interaction may include that the user terminal sends a control instruction to the controlled device, and may also include that the controlled device returns status information or returns an instruction execution result to the user terminal, and the like. The data interaction between the user terminal and the controlled device may be triggered by a client installed in the user terminal.

As an embodiment, the user terminal obtains the identifier of the target object. Specifically, the above-mentioned data set may include the product identifier corresponding to the preset identity information, and the product identifier may be the name and model of the product. After the identity information of the target object is obtained, the product identification of the target object can be determined, and an acquisition request is sent to the product server corresponding to the product identification through the above-mentioned Internet of Things. If the acquisition request includes the product identification, the product server obtains the acquisition request. Afterwards, the request is parsed to obtain a product identifier, and the product description information corresponding to the product identifier is obtained, which is used as the product description information of the target object, and is sent to the user terminal.

S303: Acquire visual presentation information of the target object in the real environment.

Wherein, the visual presentation information includes pose information of the target object relative to the image acquisition device.

S304: Generate virtual content according to the product description information and the visual presentation information, where the virtual content is used to represent the product description information.

As an implementation manner, the virtual content can reflect the product description information, that is, the user can learn the product description information by observing the virtual content. As an implementation manner, the virtual content may be an animation display, text display, or picture display of product description information, etc., which will not be repeated here. For specific implementation manners of acquiring virtual content, reference may be made to subsequent embodiments.

Therefore, the product description information is used to determine the information represented by the virtual content, that is, the information represented by the virtual content is related to the product description information, so that the product description information can be known through the information represented by the virtual content. The visual presentation information is used to reflect the information presented by the virtual content that can be observed by the user when the virtual content is placed in the real world. The information may include placement information or lighting information, and the placement information may include the virtual content. For information such as position and placement angle, specifically, for an implementation manner of determining virtual content according to visual presentation information, reference may be made to subsequent embodiments.

S305: Add the virtual content at the position of the target object in the real environment.

Specifically, the position information of the target object in the real environment is acquired in advance.

As an implementation manner, the position information of the target object in the real environment may be determined according to the image of the target object collected by the image collection device.

Specifically, the real environment may refer to the world coordinate system corresponding to the real world, and the position information of the target object in the real environment may refer to the physical coordinates of the target object in the world coordinate system. The world coordinate system may be a coordinate system established with the image acquisition device as the center, that is, the position of the image acquisition device in the real world scene is taken as the origin of the world coordinate system. Specifically, the image acquisition device is installed in this When the execution subject of the applied method, for example, when the method of the present application is a user terminal, the center of the world coordinate system is the position of the user terminal.

The user terminal can scan the surrounding environment according to the preset positioning algorithm to establish a world coordinate system centered on the terminal, and determine the coordinate position of each real object in the real environment in the world coordinate system, and the coordinate position is used as the real object. Location information in the real environment. As an embodiment, the user terminal is provided with an image acquisition device and an inertial measurement unit, and an image corresponding to the real environment can be established according to the image of the surrounding environment collected by the image acquisition device and the pose information of the user terminal obtained by the inertial measurement unit. The world coordinate system, that is, the world coordinate system centered on the terminal, and then obtain the coordinate position of each real object in the world coordinate system.

Specifically, the above-mentioned SLAM technology can be used to understand the surrounding real environment and track real objects in the real environment. The SLAM technology can be based on the image of the surrounding environment collected by the camera and the pose information of the user terminal obtained by the inertial measurement unit. , build a world coordinate system based on the terminal as the starting point, and then use the time offlight (TOF) depth camera to obtain a dense 3D point cloud, where the dense 3D point cloud can obtain the world coordinates of each point on the surface of the real object 3D coordinates under the system, the 3D coordinates can be used as the position information of each real object in the real environment.

After the position of the target object in the real environment is determined, virtual content is added to the position of the target object in the real environment. Specifically, the implementation of adding virtual content in the real environment to enable the user to observe the AR effect can refer to the above implementation. method, which will not be repeated here.

Specifically, according to the position of the target object in the real environment, the position of the added virtual content can be located at the position of the target object in the world coordinate system. Specifically, there is a mapping relationship between the location information of the virtual content in the real environment and the display information of the display content corresponding to the virtual content on the display screen of the user terminal, and the display information includes the display size of the display content corresponding to the virtual content, shape and location.

As an embodiment, in the image of the real environment collected by the camera, the real object is in the camera coordinate system corresponding to the camera, and the Z axis of the camera coordinate system matches the optical axis direction of the camera. Specifically, the optical axis direction of the camera is the Z-axis direction of the camera coordinate system, and the XOY plane formed by the X-axis and the Y-axis is perpendicular to the Z-axis. Then the coordinates of the real object in the camera coordinate system can be determined. For example, according to the mapping relationship between the pixel coordinate system of the image collected by the camera and the camera coordinate system, the coordinates of the pixels of the image of each real object in the image in the camera coordinate system can be determined, and the coordinates include the depth of field information of the real object, such as , the projection of the coordinates of the real object on the Z axis of the camera coordinate system is the depth of field information of the moving target. The change of the depth of field of the real object can be determined through the change of the coordinates of the real object in the camera coordinate system, and the distance of the real object from the camera can be determined according to the change of the depth of field.

An implementation of displaying virtual content in the real environment, the virtual content is displayed on a certain plane or the position of an object in the real environment, and the distance between the object and the camera can be determined, and then the virtual content is displayed. The corresponding depth of field information can be determined, then through the preset corresponding relationship between the depth of field information and the contour information in the display corresponding to the virtual content, the contour information of the display content corresponding to the depth of field information of the target object can be determined. The information includes the shape and size of the display content corresponding to the virtual content. For example, according to the rule that the longer the distance or the larger the depth of field information, the smaller the outline, that is, the smaller the distance is, the smaller the distance is, the correspondence between different positions in the real environment and the outline information of the displayed content corresponding to the virtual content can be preset. Further, the shape and size of the display content corresponding to the virtual content are determined.

As another embodiment, the coordinate relationship between each pixel in the pixel coordinates of the screen and each position in the real environment can also be determined. Specifically, it can be determined by the internal and external parameters of the camera. For example, it can be determined by Zhang Zhengyou's calibration method. . For example, as shown in Figure 2, the virtual content A is displayed on the ground near the desk lamp in the real environment, and the position of the ground near the desk lamp in the image of the real environment within the pixel coordinates of the screen can be determined, so that the virtual content can be determined to correspond to The display content is displayed at the position of the ground near the desk lamp in the image of the real environment displayed on the screen.

Therefore, through the predetermined mapping relationship between the pixel coordinates of each pixel of the image displayed on the screen and the world coordinates of each position in the world coordinate system, virtual content can be added to a specific position in the real environment.

For example, the virtual content needs to be displayed at the position A in the specified coordinate system, that is, when the user uses the user terminal, he can see that the virtual object is displayed at the position A in the real space, and then the corresponding position A is determined. The display position on the display screen is position B, then when the virtual object is displayed at position B on the display screen of the user terminal, the user can see a virtual object displayed at position A through the display screen.

As an embodiment, after the position of the target object is determined and the virtual content is determined according to the product description information of the target object, the virtual content may be added to the position of the target object in the real world. As shown in FIG. 4 , the content displayed on the interface includes the target object 401 and the current real environment and virtual content 402 of the target object 401 .

As an implementation manner, the content of the interface may be the content observed by the user through the above-mentioned head-mounted display device. As another implementation manner, the content of the interface may be the content displayed on the screen of the user terminal. Specifically, the target object 401 displayed on the screen and the current real environment of the target object 401 may be images of the user terminal. If the image collected by the collecting device is used, the virtual content 402 may be display content displayed on the screen of the user terminal, and the display content corresponds to the virtual content 402 .

As shown in FIG. 4 , the virtual content 402 may be the textual presentation content of the product description information of the target object. As shown in FIG. 4 , the virtual content added at the position of the target object 401 (ie, the range hood in FIG. 4 ) is two callout boxes, which respectively record the introduction and operation steps of each part of the product. Through the virtual content, the user can understand the characteristics of each part of the target object and the entire operation steps.

Exemplarily, taking FIG. 4 as an example, the execution subject of the above method is the user terminal. When in use, the user photographs the range hood through the user terminal, then the user terminal can determine the position of the range hood in the real world, and in the At the position of the range hood, virtual content corresponding to the product description information of the range hood is added, such as product introduction information and operation step information of the range hood, so as to facilitate the user to quickly understand and operate the range hood.

Please refer to FIG. 5. FIG. 5 shows a method for displaying virtual content provided by an embodiment of the present application, which is used to display virtual content at the position of a target object, so as to improve the convenience for users to read the product specification and reduce the time and cost of reading the specification. It is more tedious and more able to attract users to read the product manual and improve the convenience for users to know the content of the product manual. Specifically, the method includes: S510 to S560.

S510: Acquire an image of the real environment collected by the image collection device.

S520: If a target object exists in the image, obtain product description information corresponding to the target object.

S530: Acquire model data corresponding to the target object.

The model data may refer to the product design data of the target object. Specifically, when the developer designs the target object or designs the display effect of the target object, the developer will set the model of the target object. The data corresponding to the model may be the above-mentioned model data, and the model data includes the outline shape, size ratio, appearance color and other data of the target object. When producing the target object, the target object can be produced according to the model data.

As an implementation manner, the model data of the target object may be stored in a product server corresponding to the target object, and the user terminal may send a data acquisition request to the product server, wherein the data acquisition request includes the product identifier corresponding to the target object , the product server acquires the model data of the target object based on the product identifier corresponding to the target object in response to the data acquisition request.

As another implementation manner, the model data of the target object can be stored in the above-mentioned data processing server, the user terminal can send a data acquisition request to the data processing server, and the data processing server can return the model of the target object according to the data acquisition request data. The data processing server may synchronize with at least part of the data in the product server, so that the model data of the target object stored in the processing server is consistent with the model data in the product server.

S540: Generate a virtual model corresponding to the target object based on the model data.

The virtual model may be a virtual object that can be added to the real environment. Specifically, taking the execution subject of the present application as an example of a user terminal, the virtual model may be a corresponding target object that can be displayed on the screen of the user terminal. target image. As an implementation manner, the virtual model may be a 2D plan view or a 3D three-dimensional view.

Taking the target object 401 in FIG. 4 as an example, for example, if the target object 401 is a range hood, after obtaining the model data of the range hood, that is, the shape outline data, size and other data of the range hood, The virtual model of the range hood is generated according to the model data, that is, a virtual range hood is generated, and the graph of the virtual range hood is the same as the shape outline of the physical range hood, and the size can be the size of the physical range hood proportional scaling.

As an embodiment, the virtual model can be added to the real environment, and the user can observe the virtual model in the real environment through the AR device, that is, the virtual target object can be observed in the real environment.

S550: Generate the virtual content based on the product description information and the virtual model.

As an embodiment, the virtual model can be changed according to the product description information. For example, based on the virtual model, a specified content is added to the virtual model. The specified content is set according to the product description information, and then the specified content is added. A virtual model of the specified content can be added within the real environment as virtual content.

Specifically, the specified content can be added to the virtual model, and the label content can be a text label, and the text label is used to indicate the function, description of each part of the target object, the overall operation steps or the description of the overall product, etc. The text label The label corresponds to the product description information, and specifically, a text label may be set according to the product description information. It should be noted that the text annotation does not only include text, but may also include information in formats such as pictures or voices.

As shown in FIG. 6 , the content displayed on the interface includes the target object 401 and the current real environment and virtual content of the target object 401. Specifically, the virtual content includes a virtual model 4021 and a text annotation 4022, as shown in FIG. 6 , The target object 401 is an object represented by a dotted line, for example, a range hood, and the virtual model 4021 is represented by a realization, which is wrapped outside the target object 401 .

Referring to FIG. 4 , the method of adding the text annotation 4022 to the virtual model 4021 is also compared to the method of adding the text annotation 4022 to the target object 401 , the former can reduce the amount of calculation, because the method of adding the text annotation 4022 to the target object 401 , it is necessary to accurately find the position of each part of the target object 401 in the real environment, which will increase the overall calculation amount, and because in the embodiment of the present application, the virtual model 4021 is set according to the model data of the target object, so that the The size and scale of the virtual model 4021 can be similar to the original target object, and by manipulating the virtual model 4021, for example, adding text information on the virtual model 4021, flexibility can be increased.

As shown in FIG. 7 , the content displayed on the interface includes the target object 401 and the current real environment and virtual content of the target object 401 , and the virtual content includes a virtual model 4021 and a text annotation 4022 , wherein the text annotation 4022 may include adding The first content on the surface of the virtual model 4021 and the second content provided outside the surface of the virtual model 4021, wherein the first content and the second content may be associated with each other, as shown in FIG. 7, the first content is a number, playing a labeling role, the second content is the operation content, the operation content corresponds to the number, as shown in Figure 7 "Remove the **** at 1", therefore, the user combines the first content and the second The content can make it clear how to operate the target object without having to consult the instruction manual of the target object.

As an implementation manner, the product description may be displayed in front of the user with an AR effect in an animation manner, and an implementation manner of generating the virtual content based on the product description information and the virtual model may be based on the The product description information and the virtual model are used to generate a dynamic display image, and the dynamic display image is used as the virtual content.

Specifically, the dynamic presentation picture may be a dynamic picture established based on the virtual model. As an embodiment, the product description information may be product part introduction information, and the dynamic display screen may be a virtual model that rotates in a predetermined order, and multiple parts of the virtual model are displayed in the user's field of vision by rotating the virtual model. Within the range, and the description information of the part is displayed at each part, at the same time, the shell of the virtual model can also be opened, so that the internal structure of the virtual model can be displayed.

As another embodiment, the product description information may be product operation information, and the operation information includes information that the target object needs to be operated in a specified order to complete specific operations and functions, such as installation, disassembly, and cleaning of the product. Then, the dynamic display screen may be a dynamic display of the operation process corresponding to the operation information. For example, if the operation information is the dismantling operation of the target object, the dynamic display picture may be a process display animation in which various parts of the virtual model of the target object are dismantled sequentially in a specified order.

As an embodiment, information such as the pose, lighting, etc. of the target object in the real environment can also be obtained, and a virtual model is determined based on the information, so that the virtual model can be closer to the placement position of the target object in the real environment. Specifically, See subsequent examples.

S560: Add the virtual content at the position of the target object in the real environment.

Please refer to FIG. 8 . FIG. 8 shows a method for displaying virtual content provided by an embodiment of the present application, which is used to display virtual content at the position of a target object, so as to improve the convenience for users to read the product manual and reduce the time spent in reading the manual. It is more tedious and more able to attract users to read the product manual and improve the convenience for users to know the content of the product manual. Specifically, the method includes: S810 to S870.

S810: Acquire an image of the real environment collected by the image collection device.

S820: If a target object exists in the image, obtain product description information corresponding to the target object.

S830: Acquire model data corresponding to the target object.

S840: Acquire visual presentation information of the target object in the real environment.

The visual presentation information is used to represent the user's visual perception of the target object observed in the real environment, and specifically, may be the placement angle, illumination, and other phenomena of the observed target object.

As an embodiment, the visual presentation information includes at least one of illumination information of the target object and pose information of the target object relative to the image acquisition device. As an embodiment, the method is applied to an electronic device, and the image acquisition device is installed in the electronic device, and the pose information of the target object relative to the image capture device is also equivalent to the pose information of the target object relative to the electronic device.

The pose information of the target object in the real environment may include information such as the position and rotation angle of the target object in the real environment. Specifically, the pose information is the position and rotation information between the target object and the image acquisition device. There may be one or more target objects in the collected target image. When there are multiple target objects in the collected target image, the attitude information between each target object in the target image and the image acquisition device is acquired.

The manner of obtaining the pose information may be obtained by using markers preset on the target object, or may be determined according to the image of the target object according to a pre-trained analysis model.

As an embodiment, the target object is provided with one or more markers, each marker includes a plurality of mutually separated sub-markers, and each sub-marker includes one or more feature points, wherein the markers The marker includes a background area and several sub-markers distributed in the background area according to specific rules, and each sub-marker has one or more feature points.

The sub-marker is a pattern with a certain shape, and the color of the sub-marker has a certain degree of distinction from the color of the background area in the marker. For example, the background area is white, and the color of the sub-marker is black. The sub-marker may be composed of one or more feature points, and the shape of the feature point is not limited, and may be a dot, a ring, a triangle, or other shapes.

Specifically, the pixel coordinates of the feature points in the image in the image coordinate system corresponding to the image are acquired, and the image is acquired according to the pixel coordinates of the feature points in the image and the pre-acquired physical coordinates corresponding to the feature points. The attitude information between the device and the target object is collected. Wherein, the feature point may be the above-mentioned sub-marker.

The physical coordinates are the coordinates of the pre-acquired feature points in the physical coordinate system corresponding to the target object, and the physical coordinates of the feature points are the real positions of the feature points on the corresponding target object. The physical coordinates of each feature point can be acquired in advance. Specifically, multiple feature points and multiple markers are set on the marking surface of the target object, and a certain point on the marking surface is selected as the origin to establish a physical coordinate system. Take the marked surface as the XOY plane of the physical coordinate system, and the origin of the XOY coordinate system is located in the marked surface. Wherein, the physical coordinates of each feature point may be determined in advance according to the position of each feature point on the marker and the distance from a certain reference point on the marker, and then the center of the physical coordinate system may be the reference point .

When the pose of the target object is different, in the collected image of the target object, the distance between the sub-marker in the marker on the target object and the reference point and the position of the sub-marker are different. Therefore, Through the mapping relationship between the pixel coordinates and the physical coordinates, the position of each sub-marker in the real environment can be determined, so that the position of the marker can be determined as a whole, and then the position of the target object can be determined according to the installation position of the marker on the target object. posture.

In other embodiments, the visual presentation information may include illumination information of the target object and pose information of the target object relative to the image capturing device. The electronic device may be the execution body of this embodiment of the method, for example, may be a user terminal. The lighting information may be the light and shadow effects of the target object in the real environment, for example, the light and dark effects of the surface of the target object and the shadow of the target object in the real environment.

As another implementation manner, the pose information may also be determined according to a pre-trained analysis model, specifically, please refer to FIG. 9 , the S840 includes: S841 to S843.

S841: Acquire a plurality of sample data, where the sample data is image data obtained in advance according to model data corresponding to a plurality of different visual presentation information of the target object.

In the embodiment of the present application, an analysis model can be pre-trained, and the analysis model can obtain the visual presentation information of the target object based on the image analysis of the target object collected by the image acquisition device.

As an embodiment, the analysis model may be a neural network-based machine learning model. Specifically, the analysis model may be a convolutional neural network (Convolutional Neural Networks, CNN). The analysis model in the embodiment of the present application is set based on CNN. Specifically, the analysis model is shown in FIG. 10 . The image collected by the image acquisition device is input into the convolutional neural network, and the convolutional neural network inputs the feature vector of the image into the first A fully-connected layer and a second fully-connected layer, wherein the first fully-connected layer outputs a detection result, and the detection result is used to determine whether a target object exists in the image of the real environment collected by the image acquisition device, and to determine the product identification of the target object. The second fully connected layer outputs visual presentation information.

Among them, the method of directly regressing the pose through the end-to-end CNN network, such as posenet, usually has an order of magnitude lower accuracy than the traditional feature point matching method. This is due to the way CNNs, more similar to image retrieval, are affected by the training data.

Therefore, in order to improve the accuracy of the neural network, the embodiment of the present application can render richer sample data by using the model data of the target object. Moreover, the analysis model of the present application can also complete the detection task at the same time, without requiring an external detection module.

Specifically, the application of the analysis model of the present application includes two parts, namely, an offline training part and an online running part. As shown in FIG. 11 , in the offline training part, the rendering model is used to generate sample data, and specifically, it is used to input the 3D model, pose, and lighting parameters, and output the image projected from the model to 2D. The image projected to the 2D may be a virtual model corresponding to the target object generated based on the model data.

Specifically, the model data corresponding to the target object is obtained, the pose and lighting parameters corresponding to the model data are set, and the model data, pose and lighting parameters are input into the rendering model, and the rendering model can be based on the model data and pose and illumination parameters to output a two-dimensional image of the target object, wherein, the two-dimensional image can be regarded as the image of the target object collected by the image acquisition device in the real environment of the pose and illumination parameters corresponding to the model data, That is, the two-dimensional image can represent the image of the target object under the specified pose and illumination.

Specifically, the rendering model may be graphics design software or an application program with graphics processing function, which can convert 3D graphics into 2D graphics, for example, a three-dimensional graphics rendering tool, which can render the rendering model based on model data, pose and lighting parameters. A 3D object in pose and lighting, and converted to a 2D image of the object.

Then, based on the model data of the target object and a plurality of different input poses and illuminations, a variety of different image data under different poses and illuminations, that is, the above-mentioned two-dimensional images can be obtained.

S842: Train an analysis model based on a plurality of the sample data and each of the corresponding visual presentation information.

Then, the plurality of sample data is input into an analysis model, and the analysis model is trained based on the plurality of the sample data and each of the corresponding visual presentation information. Among them, the pose and illumination parameters corresponding to each sample data are used as the true value of the sample data, that is, the label, so that the model is trained according to the sample and the label of each sample, so that the model can analyze the image of the target object to obtain the target object in A function of visually presenting information within the real environment.

Specifically, the detection result is evaluated using the cross-entropy error, and the result of visually presenting the information is evaluated using the minimum mean error. The error direction is passed, and the parameters of the model are optimized using the gradient descent method.

S843: Analyze the image including the target object based on the trained analysis model to obtain visual presentation information of the target object in the real environment.

Please refer to FIG. 11, the online operation part, the image collected by the image acquisition device, that is, the camera image in FIG. 11 is input into the analysis model, and the analysis model outputs the result of the image, referring to the process of FIG. 10, to obtain the detection result and visual presentation information, According to the detection result, it is determined whether there is a target object in the image, and if there is, the visual presentation information is input into the rendering module to obtain a virtual model corresponding to the target object.

S850: Generate a virtual model corresponding to the target object based on the model data and the visual presentation information.

In some embodiments, the visual presentation information includes pose information, and generating a virtual model corresponding to the target object based on the model data and the visual presentation information includes: generating based on the model data and the pose information The virtual model corresponding to the target object, where the virtual model and the target object have the same pose.

As an implementation manner, model data corresponding to different position information is pre-determined, among which, the pose information may include position and rotation information between the target object and the image acquisition device. Specifically, the rotation information can be used as the orientation of the target object in the real environment. According to the position between the target object and the image acquisition device, the distance between the target object and the image acquisition device in the real environment can be obtained. the distance.

In some embodiments, the model data includes size information and a pose, and the pose can be used as an orientation of the virtual model, for example, a front view or a rear view. Then, the size information is determined according to the position between the target object and the image acquisition device. For example, the closer the distance, the larger the size, and the further the distance, the smaller the size. Determine the placement posture according to the rotation information between the target object and the image acquisition device, use the determined size information and placement posture as model data, and generate a virtual model according to the model data, so that the size of the virtual model is not smaller than The size of the target object in the image of the real environment collected by the image acquisition device, and the placement position of the virtual model is the same as the placement pose of the target object in the real environment.

Specifically, the model data corresponding to the target object includes a plurality of size information and placement positions, and each size information corresponds to a position, and each placement pose corresponds to rotation information. Therefore, after the target object and the image are acquired After the position and rotation information between the devices is collected, the size information corresponding to the position is searched, and the placement posture corresponding to the rotation information is searched.

S860: Generate the virtual content based on the product description information and the virtual model.

S870: Add the virtual content at the position of the target object in the real environment.

After referring to the pose information, in the virtual content observed by the user, the virtual model of the target object moves with the movement of the user's viewing direction, position, and viewing angle, that is, the virtual content changes according to the user's viewing angle. The same as the target object that the user sees.

As shown in FIG. 12 , compared with FIGS. 6 and 7 , it can be seen that after the observation angle of the image capture device changes, the virtual model changes with the change of the pose of the target object in the image, so that the user can communicate with the user through the image capture device. In the AR effect seen on the screen of the terminal, the virtual model changes with the change of the pose of the target object in the image, which is more realistic.

As an embodiment, the virtual model is a virtual model of the target object, and the virtual model matches the size of the target object in the image. Specifically, the size of the virtual model and the target object in the real environment is matched. The positions are the same, and the virtual model and the target object have the same size, so that the virtual model and the target object can be completely coincident, so that the user can observe that the target object is completely replaced by the virtual model.

Therefore, the user can photograph the target object through the user terminal, and display virtual content at the position of the target object, and as the viewing angle and position of the user terminal and the user change, the virtual content also changes accordingly.

Please refer to FIG. 13 , which shows an apparatus 900 for displaying virtual content provided by an embodiment of the present application. The apparatus may include: a first acquiring unit 1301 , a second acquiring unit 1302 , a determining unit 1303 and a processing unit 1304 .

The first obtaining unit 1301 is configured to obtain an image of the real environment collected by the image collecting device.

The second acquiring unit 1302 is configured to acquire, if a target object exists in the image, the product description information corresponding to the target object and the visual presentation information of the target object in the real environment, where the visual presentation information includes all pose information of the target object relative to the image acquisition device.

The determining unit 1303 is configured to generate virtual content according to the product description information and the visual presentation information, where the virtual content is used to represent the product description information.

The processing unit 1304 is configured to add the virtual content at the position of the target object in the real environment.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices and modules, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

Please refer to FIG. 14 , which shows an apparatus 1400 for displaying virtual content provided by an embodiment of the present application. The apparatus may include: a first acquiring unit 1410 , a second acquiring unit 1420 , a determining unit 1430 and a processing unit 1440 .

The first obtaining unit 1410 is configured to obtain an image of the real environment collected by the image collecting device.

The second obtaining unit 1420 is configured to obtain product description information corresponding to the target object if there is a target object in the image.

The determining unit 1430 is configured to generate virtual content according to the product description information and the visual presentation information, where the virtual content is used to represent the product description information.

The determination unit 1430 includes an acquisition subunit 1431 , a model subunit 1432 and a content subunit 1433 .

The acquiring subunit 1431 is used to acquire model data corresponding to the target object.

The model subunit 1432 is configured to generate a virtual model corresponding to the target object based on the model data and the visual presentation information.

Specifically, the model subunit 1432 is configured to obtain the visual presentation information of the target object in the real environment, where the visual presentation information includes illumination information of the target object and the relative information of the target object relative to the image acquisition device. At least one type of pose information; generating a virtual model corresponding to the target object based on the model data and the visual presentation information.

Further, the visual presentation information includes the pose information, and the model subunit 1432 is configured to generate a virtual model corresponding to the target object based on the model data and the pose information, the virtual model and the target The objects are placed in the same pose.

Further, the virtual model is a virtual model of the target object, and the virtual model matches the size of the target object in the image.

Further, the visual presentation information includes the pose information, and the model subunit 1432 is used to acquire a plurality of sample data, and the sample data is obtained in advance according to the model data corresponding to a plurality of different visual presentation information of the target object. based on a plurality of the sample data and each of the corresponding visual presentation information, train an analysis model; analyze the image including the target object based on the trained analysis model, and obtain the target object in the Describe the visual presentation of information in the real environment.

The content subunit 1433 is configured to generate the virtual content based on the product description information and the virtual model.

Specifically, the content subunit 1433 is configured to generate a dynamic display image based on the product description information and the virtual model, and the dynamic display image is used as the virtual content.

The processing unit 1440 is configured to add the virtual content at the position of the target object in the real environment.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices and modules, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In several embodiments provided in this application, the coupling between the modules may be electrical, mechanical or other forms of coupling.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically alone, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules.

Please refer to FIG. 15 , which shows a structural block diagram of an electronic device provided by an embodiment of the present application. The electronic device 1500 may be an electronic device capable of running an application program, such as a smart phone, a tablet computer, an electronic book, or the like. Specifically, the electronic device may be the above-mentioned head-mounted display or a user terminal.

The electronic device 1500 in the present application may include one or more of the following components: a processor 1510, a memory 1520, and one or more application programs, wherein the one or more application programs may be stored in the memory 1520 and configured by One or more processors 1510 execute, and one or more programs are configured to perform the methods described in the foregoing method embodiments.

The processor 1510 may include one or more processing cores. The processor 1510 uses various interfaces and lines to connect various parts of the entire electronic device 1500, and executes by running or executing the instructions, programs, code sets or instruction sets stored in the memory 1520, and calling the data stored in the memory 1520. Various functions of the electronic device 1500 and processing data. Optionally, the processor 1510 may use at least one of a digital signal processing (Digital Signal Processing, DSP), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and a Programmable Logic Array (Programmable Logic Array, PLA). implemented in a hardware form. The processor 1510 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly handles the operating system, user interface and application programs, etc.; the GPU is used for rendering and drawing of the display content; the modem is used to handle wireless communication. It can be understood that, the above-mentioned modem may not be integrated into the processor 1510, and is implemented by a communication chip alone.

The memory 1520 may include random access memory (Random Access Memory, RAM), or may include read-only memory (Read-Only Memory). Memory 1520 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 1520 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing the following method embodiments, and the like. The storage data area may also store data created by the terminal 100 during use (such as phone book, audio and video data, chat record data) and the like.

Please refer to FIG. 16 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. The computer-readable medium 1600 stores program codes, and the program codes can be invoked by the processor to execute the methods described in the above method embodiments.

The computer-readable storage medium 1600 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 1600 includes a non-transitory computer-readable storage medium. Computer readable storage medium 1600 has storage space for program code 1610 to perform any of the method steps in the above-described methods. These program codes can be read from or written to one or more computer program products. Program code 1610 may be compressed, for example, in a suitable form.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or some technical features thereof are equivalently replaced; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (10)

1. A method for displaying virtual content, comprising: acquiring an image of the real environment captured by the image capturing device; If there is a target object in the image, obtain product description information corresponding to the target object; acquiring visual presentation information of the target object in the real environment, where the visual presentation information includes pose information of the target object relative to the image acquisition device; generating virtual content according to the product description information and the visual presentation information, where the virtual content is used to represent the product description information; The virtual content is added at the location of the target object in the real environment. 2. The method according to claim 1, wherein the generating virtual content according to the product description information and the visual presentation information comprises: obtaining model data corresponding to the target object; generating a virtual model corresponding to the target object based on the model data and the visual presentation information; The virtual content is generated based on the product description information and the virtual model. 3. The method according to claim 2, wherein the generating the virtual content based on the product description information and the virtual model comprises: A dynamic display image is generated based on the product description information and the virtual model, and the dynamic display image is used as the virtual content. 4 . The method according to claim 2 , wherein the model data includes size information and a pose, and the pose information includes position and rotation information between the target object and the image acquisition device, 5 . The generating a virtual model corresponding to the target object based on the model data and the visual presentation information includes: Determine the size information according to the position between the target object and the image acquisition device, and determine the placement posture according to the rotation information between the target object and the image acquisition device; The virtual model is acquired according to the determined size information and the determined placement position. 5. The method according to claim 2, wherein the generating the virtual content based on the product description information and the virtual model comprises: Designated content is added to the virtual content, the designated content is set based on the product description information, and a virtual model of the designated content is added as the virtual content. 6 . The method according to claim 2 , wherein the virtual model is a virtual model of the target object, and the virtual model matches the size of the target object in the image. 7 . 7. The method according to claim 1, wherein the acquiring visual presentation information of the target object in the real environment comprises: acquiring a plurality of sample data, the sample data being image data obtained in advance according to model data corresponding to a plurality of different visual presentation information of the target object; training an analysis model based on a plurality of the sample data and each of the visual presentation information; Based on the trained analysis model, the image including the target object is analyzed, and the visual presentation information of the target object in the real environment is obtained. 8. A display device for virtual content, comprising: a first acquiring unit, configured to acquire an image of the real environment acquired by the image acquisition device; The second acquiring unit is configured to acquire, if there is a target object in the image, the product description information corresponding to the target object and the visual presentation information of the target object in the real environment, where the visual presentation information includes the pose information of the target object relative to the image acquisition device; a determining unit, configured to generate virtual content according to the product description information and the visual presentation information, where the virtual content is used to represent the product description information; A processing unit, configured to add the virtual content at the position of the target object in the real environment. 9. An electronic device, characterized in that, comprising: one or more processors; memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more application programs are configured to execute The method of any one of claims 1-7. 10. A computer-readable medium, characterized in that the computer-readable medium stores program codes executable by a processor, and when the program codes are executed by the processor, the processor executes claims 1- 7 any one of the methods.

Images