CN112039937A - A display method, position determination method and device - Google Patents

Abstract

The embodiment of the application discloses a display method, which comprises the following steps: the first terminal device sends a first image containing the target object to the second terminal device, and the second terminal device sends first position information of a position in the first image corresponding to the operation of the first position on the target object to the first terminal device. And the first terminal equipment determines that the first position on the target object corresponds to a second position in the virtual coordinate system according to the first position information, and the second position corresponds to the actual position of the first position on the target object in the real three-dimensional space. After determining the second position, a marker may be added at the second position in the virtual coordinate system and rendered on a second image including the target object displayed on the first terminal device such that a position of the marker on the second image corresponds to a position of the first position on the target object in the second image, and the marker rendered on the second image is displayed.

Description

A display method, position determination method and device

technical field

The present application relates to the field of computers, and in particular, to a display method, a position determination method, and an apparatus.

Background technique

With the development of science and technology, remote assistance functions have become commonplace. The remote assistance function based on augmented reality (AR) technology is favored by more and more users because it can realize a user experience similar to "hands-on" remote assistance.

It can be understood with reference to FIG. 1 , which is a schematic diagram of an exemplary scenario of remote assistance provided by an embodiment of the present application. The scenario shown in FIG. 1 is that user A (not shown in the figure) and user B (not shown in the figure) are in a video call. During the video call, user A enables the remote assistance function. After the terminal device 110 used by user A sends the captured image 101 to the terminal device 120 used by user B, the terminal device 120 can display the image 101, and user B can perform corresponding operations on the terminal device 120, such as clicking on the terminal device 120 The screen position on the screen corresponding to the first position on the target object in the image 101 , eg, point P on the cylinder shown in FIG. 1 . The terminal device 120 sends the screen position information corresponding to point P, such as screen coordinates, to the terminal device 110, and the terminal device 110 renders an auxiliary mark at the screen position described by the screen position information corresponding to point P, to prompt the user A to perform corresponding actions based on the auxiliary mark. operation.

It can be understood that, in practical applications, there is a certain time difference between the time when the terminal device 110 sends the image 101 to the terminal device 120 and the time when the terminal device 120 sends the screen position corresponding to point P to the terminal device 110 . Because it takes a certain amount of time for the terminal device 110 to send the image 101 to the terminal device 120, it takes a certain amount of time for the user B to perform the corresponding operation on the terminal device 120, and the terminal device 120 sends the screen position information corresponding to point P to the terminal device 110. It also takes some time. During this time difference, the position of the terminal device 110 may have changed, because user A may move or shake the terminal device during the video call, etc., which causes the terminal device 110 to receive the screen position information. The image 102 captured by the device 110 is not the same as the image 101 . That is, the screen position described by the screen position information received by the terminal device 110 is inconsistent with the first position on the current target object, that is, the position of point P in the image 102 . As a result, the position of the auxiliary marker rendered by the terminal device 110 based on the screen position described by the aforementioned screen position information (the position of the Q point in FIG. 1 ) is inconsistent with the position of the P point on the target object in the image 102, so that the remote assistance The effect of the function is not good, and the real "hands-on" remote assistance cannot be realized.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a display method and device, which can solve the problem that the traditional remote assistance function based on AR technology cannot realize the real "hands-on" remote assistance.

In a first aspect, an embodiment of the present application provides a display method. Specifically, a first terminal device may send a first image containing a target object to a second terminal device, and after the second terminal device receives the first image, The first image can be displayed, the user using the second terminal device can trigger an operation for the first position on the target object on the second terminal device, and the second terminal device will correspond to the user targeting on the target object on the second terminal device. The operation of the first position corresponds to the first position information of the position in the first image, and is sent to the first terminal device. Considering that even if the position of the first terminal device changes after sending the first image to the second terminal device, the first position on the target object corresponding to the operation triggered by the user on the second terminal is in the real three-dimensional space. The actual location does not change. Therefore, in this embodiment of the present application, the first terminal device may determine, according to the aforementioned first position information, that the first position on the target object corresponds to the second position in the virtual coordinate system, and the second position in the virtual coordinate system corresponds to The actual position of the first position on the target object in the real three-dimensional space. After it is determined that the first position on the target object corresponds to the second position in the virtual coordinate system, a mark may be added at the second position in the virtual coordinate system, and the second position including the target object displayed on the first terminal device. Render the mark on the image, so that the corresponding position of the mark on the second image is consistent with the position of the first position on the target object in the second image, and display the image rendered on the second image. the mark. After the first terminal device displays the aforementioned mark rendered on the second image, the user using the first terminal device can perform further operations based on the mark, thereby achieving the effect of "hands-on" remote assistance.

In a possible implementation manner, after the first terminal device sends the first image to the second terminal device, the second terminal device may display the first image. Then, the user using the second terminal device can trigger an operation for the first position on the target object on the screen of the second terminal device, and the second terminal device can obtain the user's target on the screen of the second terminal device for the aforementioned target object. The screen position corresponding to the operation of the first position, and the first position information is determined according to the screen position. and send the first position information to the first terminal device, so that the first terminal device can determine, based on the first position information, that the first position on the target object corresponds to the second position in the virtual coordinate axis, so that in the virtual coordinate system A mark is added at the second position in the first terminal device, and the mark is rendered on the second image including the target object displayed on the first terminal device, so that the position corresponding to the mark on the second image is the same as that on the target object. The first position is consistent with the position in the second image and displays the marker rendered on the second image. After the first terminal device displays the aforementioned mark rendered on the second image, the user using the first terminal device can perform further operations based on the mark, thereby achieving the effect of "hands-on" remote assistance.

In a possible implementation manner, considering the practical application, to determine the absolute image position information of the first position on the target object in the first image, the image resolution of the first image and the target The relative image position of the first position on the object in the first image is determined, wherein the image resolution of the first image can be sent by the first terminal device to the second terminal device. In order to reduce the amount of data sent by the first terminal device to the second terminal device, the first position information may be the aforementioned information describing the relative image position of the first position on the target object in the first image. In this case, the second terminal device may determine the first position information according to the aforementioned screen position and the position of the display area of the first image on the screen of the second terminal device.

In a possible implementation manner, considering that the first position on the target object corresponds to the second position in the virtual coordinate system and is related to the image resolution of the first image, in this embodiment of the present application, the first terminal The device may determine that the first position on the target object corresponds to the second position in the virtual coordinate system according to the first position information and the image resolution of the first image. Specifically, the image position of the first position on the target object in the image coordinate system corresponding to the first image is determined, and the second position in the virtual coordinate system corresponding to the first position on the target object is determined according to the aforementioned image position.

In a possible implementation manner, considering that the first position on the target object corresponds to the second position in the virtual coordinate system, the field of view information corresponding to the first image and the first position on the target object are in the first The positions of the images in the images are related, so in this embodiment of the present application, the first terminal device The first position corresponds to the second position in the virtual coordinate system. Specifically, the first terminal device may acquire the field of view information corresponding to the first image, and calculate the position of the target ray in the virtual coordinate system according to the image position and the field of view information corresponding to the first image; the target ray is that the endpoint is the target position, and A ray passing through the second location in the virtual coordinate system. Then, the first terminal device determines the intersection position of the target ray and the virtual object in the virtual coordinate system as the first position on the target object corresponding to the second position in the virtual coordinate system.

In a second aspect, an embodiment of the present application provides a method for determining a location. Specifically, a second terminal device receives a first image sent by a first terminal device, and displays the first image on a screen of the second terminal device. image; the first image is an image including a target object; the second terminal device receives an operation instruction triggered by the user on the second terminal device for the first position on the target object, and determines the first a position information; the first position information corresponds to the user's operation on the second terminal device for the first position on the target object corresponding to the position in the first image; the second terminal The device determines, according to the first position information, that the first position on the target object corresponds to the second position in the virtual coordinate system, and corresponds the first position on the target object to the second position in the virtual coordinate system Sent to the first terminal device; the second position in the virtual coordinate system corresponds to the actual position of the first position on the target object in the real three-dimensional space. After the second terminal device sends the first position on the target object corresponding to the second position in the virtual coordinate system to the first terminal device, the first terminal device may add a mark at the second position in the virtual coordinate system, and add a mark at the second position in the virtual coordinate system. Rendering the mark on the second image including the target object displayed on the first terminal device, so that the corresponding position of the mark on the second image and the first position on the target object are in the second image The first terminal device may also display the mark rendered on the second image. After the first terminal device displays the aforementioned mark rendered on the second image, the user using the first terminal device can perform further operations based on the mark, thereby achieving the effect of "hands-on" remote assistance.

In a possible implementation manner, the first position information describes the relative position of the first position on the target object in the first image; the method further includes: the second terminal device, according to The screen position corresponding to the operation on the first position on the target object triggered by the user on the second terminal device, and the display of the first image on the screen of the second terminal device The location of the area determines the first location information.

In a possible implementation manner, the second terminal device determines that the first position on the target object corresponds to the second position in the virtual coordinate system according to the first position information, including: the second terminal The device receives the image resolution of the first image sent by the first terminal device; the second terminal device determines the first image resolution on the target object according to the first position information and the image resolution of the first image The image position of a position in the image coordinate system corresponding to the first image; the second terminal device determines according to the image position that the first position on the target object corresponds to the second position in the virtual coordinate system.

In a possible implementation manner, the method further includes: receiving, by the second terminal device, information related to the virtual coordinate system sent by the first terminal device, and the information related to the virtual coordinate system information, including the position of the virtual object corresponding to the target object in the virtual coordinate system; the position of the virtual object corresponding to the target object in the virtual coordinate system, corresponding to the target object in the real three-dimensional space actual location.

In a possible implementation manner, the second terminal device determining, according to the image position, that the first position on the target object corresponds to the second position in the virtual coordinate system, including: the second terminal device receiving The field of view information corresponding to the first image sent by the first terminal device, and the field of view information corresponding to the first image is used to describe that when the first image is captured, the camera that captured the first image is at the virtual coordinates The target position in the system and the target shooting direction of the camera that shoots the first image in the virtual coordinate system; the second terminal device calculates the target ray according to the field of view information and the image position the position in the virtual coordinate system; the target ray is a ray whose endpoint is the target position and passes through the second position in the virtual coordinate system; the second terminal device converts the target ray The intersection position with the virtual object in the virtual coordinate system is determined as the first position on the target object corresponding to the second position in the virtual coordinate system.

In a third aspect, an embodiment of the present application provides a method for determining a location. Specifically, a second terminal device sends a screen capture request to a first terminal device; the second terminal device receives a first image sent by the first terminal device, and The first image is displayed on the screen of the second terminal device; the first image is an image including a target object; the second terminal device receives a trigger triggered by a user on the second terminal device for the target object the operation instruction of the first position on the target object, and determine the first position information; the second terminal device sends the first position information to the first terminal device. It can be seen that during the period when the user on the second terminal device side triggers the mark, the second terminal device statically displays the first image on the main interface of the second terminal device. When the user triggers the first position of the target object, the reference is the target object displayed in the first image. In this way, not only can the user accurately identify the position of the mark, but also the position triggered by the user can be accurately associated with the first position of the target object on the first image. Correspondingly, when the first terminal device renders the mark on the second image including the target object displayed on the first terminal device, the position of the mark displayed on the second image can accurately correspond to the object that the user wants to mark. This can improve the performance of "hands-on" remote assistance and optimize the effect of "hands-on" remote assistance.

In a possible implementation manner, the method further includes: receiving, by the second terminal device, field-of-view information corresponding to the first image, and receiving, by the second terminal device, a trigger triggered by a user on the second terminal device . The operation instruction for the first position on the target object, and determining the first position information includes: the second terminal device determines third position information according to the operation instruction, and the second terminal determines the third position information according to the third position information. The position information and the field of view information corresponding to the first image determine the first position information. The content displayed by the second terminal device comes from the first terminal device, the posture of the first terminal device has an influence on the position of the target object displayed by the first image, and the posture of the first terminal device may change frequently, so the first image The corresponding field of view information will change. The first image is an image obtained after the object in the three-dimensional space is converted into the two-dimensional space. Based on this, the second terminal device can obtain partial information as the first position information according to the field of view information corresponding to the first image and the position information triggered by the user, thereby reducing the amount of data for subsequent calculation and saving resources.

In a fourth aspect, an embodiment of the present application provides a terminal device, the terminal device includes: a first sending unit configured to send a first image to a second terminal device, where the first image is an image including a target object; a first receiving unit, configured to receive first location information from the second terminal device; the first location information corresponds to a user's operation on the second terminal device for the first location on the target object the position in the first image; a first determining unit, configured to determine, according to the first position information, that the first position on the target object corresponds to the second position in the virtual coordinate system; the virtual coordinate system The second position in the target object corresponds to the actual position of the first position on the target object in the real three-dimensional space; the adding unit is used to add a mark at the second position in the virtual coordinate system; the rendering unit, using rendering the mark on the second image, so that the corresponding position of the mark on the second image is consistent with the position of the first position on the target object in the second image; the second The image is an image including the target object displayed on the terminal device; a display unit is configured to display the mark rendered on the second image.

In a possible implementation manner, the first position information is determined according to a screen position corresponding to an operation triggered by the user on the second terminal device and targeting the first position on the target object; the first position information The first image is displayed on the screen of the two terminal devices.

In a possible implementation manner, the first position information describes the relative position of the first position on the target object in the first image; the first position information is based on the screen position, the The position of the display area of the first image on the screen of the second terminal device is determined.

In a possible implementation manner, the first determining unit is specifically configured to: determine, according to the first position information and the image resolution of the first image, where the first position on the target object is at the image position in the image coordinate system corresponding to the first image;

It is determined according to the image position that the first position on the target object corresponds to the second position in the virtual coordinate system.

In a possible implementation manner, the first determining unit is specifically configured to: acquire the field of view information corresponding to the first image, where the field of view information corresponding to the first image is used to describe when the first image is captured, the target position of the camera that captures the first image in the virtual coordinate system and the target shooting direction of the camera that captures the first image in the virtual coordinate system; according to the field of view information and the the image position, calculate the position of the target ray in the virtual coordinate system; the target ray is a ray whose endpoint is the target position and passes through the second position in the virtual coordinate system; The intersection position of the ray and the virtual object in the virtual coordinate system is determined as the first position on the target object corresponding to the second position in the virtual coordinate system; the position of the virtual object in the virtual coordinate system , corresponding to the actual position of the target object in the real three-dimensional space.

In a fifth aspect, an embodiment of the present application provides a terminal device, where the terminal device includes: a second receiving unit configured to receive a first image sent by a first terminal device, and display the image on a screen of the terminal device the first image; the first image is an image including a target object; a third receiving unit is configured to receive an operation instruction triggered by the user on the terminal device for the first position on the target object; 2. A determining unit, configured to determine first position information; the first position information corresponds to the position in the first image corresponding to the user's operation on the terminal device for the first position on the target object ; a third determining unit for determining, according to the first position information, a first position on the target object corresponding to a second position in the virtual coordinate system; a second sending unit for sending the first position on the target object The first position corresponds to the second position in the virtual coordinate system and is sent to the first terminal device; the second position in the virtual coordinate system corresponds to the first position on the target object in the real three-dimensional space. actual location.

In a possible implementation manner, the first position information describes the relative position of the first position on the target object in the first image; the apparatus further includes: a fourth determining unit, configured to Determine the screen position corresponding to the operation on the first position on the target object triggered by the user on the terminal device, and the position of the display area of the first image on the screen of the terminal device the first location information.

In a possible implementation manner, the third determining unit is specifically configured to: receive the image resolution of the first image sent by the first terminal device; according to the first position information and the first image determine the image position of the first position on the target object in the image coordinate system corresponding to the first image; according to the image position, determine that the first position on the target object corresponds to the virtual coordinates second position in the system.

In a possible implementation manner, the apparatus further includes: a fourth receiving unit, configured to receive information related to the virtual coordinate system sent by the first terminal device, the information related to the virtual coordinate system information, including the position of the virtual object corresponding to the target object in the virtual coordinate system; the position of the virtual object corresponding to the target object in the virtual coordinate system, corresponding to the target object in the real three-dimensional space actual location in .

In a possible implementation manner, the third determining unit is specifically configured to: receive visual field information corresponding to the first image sent by the first terminal device, where the visual field information corresponding to the first image is used to describe When capturing the first image, the target position of the camera that captures the first image in the virtual coordinate system and the target capture direction of the camera that captures the first image in the virtual coordinate system ; According to the visual field information and the image position, calculate the position of the target ray in the virtual coordinate system; the target ray is that the endpoint is the target position and passes through the second in the virtual coordinate system. The ray of the position; the intersection position of the target ray and the virtual object in the virtual coordinate system is determined as the first position on the target object corresponding to the second position in the virtual coordinate system.

In a sixth aspect, an embodiment of the present application provides an electronic device, the electronic device includes: a memory and at least one processor; the memory is used to store instructions; the at least one processor is used to execute the memory The instruction in , executes the method described in any one of the first aspect above.

In a seventh aspect, an embodiment of the present application provides an electronic device, the device includes: a memory and at least one processor; the memory is used to store instructions; the at least one processor is used to execute the memory in the memory. , and execute the method described in any one of the second aspect and the third aspect above.

In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, which, when executed on a computer, cause the computer to execute the method described in any one of the first aspects above.

In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, which, when executed on a computer, cause the computer to execute the method described in any one of the second aspect and the third aspect.

In a tenth aspect, an embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, causes the computer to execute the method described in any one of the first aspects above.

In an eleventh aspect, the embodiments of the present application provide a computer program product including instructions, which, when executed on a computer, causes the computer to execute the method described in any one of the second aspect and the third aspect.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments described in this application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a schematic diagram of an exemplary scenario of a remote assistance provided by an embodiment of the present application;

FIG. 2 is a schematic flowchart of a display method provided by an embodiment of the present application;

3 is a schematic diagram of a screen coordinate system provided by an embodiment of the present application;

4 is a schematic flowchart of a method for determining that a first position on a target object corresponds to a second position in a virtual coordinate system provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of a vertebral body according to an embodiment of the present application;

6 is a schematic flowchart of a method for determining that a first position on a target object corresponds to a second position in a virtual coordinate system provided by an embodiment of the present application;

7 is a schematic flowchart of a method for determining a location provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a terminal device according to an embodiment of the present application;

FIG. 9 is a schematic structural diagram of a terminal device according to an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a terminal device according to an embodiment of the present application;

FIG. 11A is a schematic diagram of an exemplary scenario of another remote assistance provided by an embodiment of the present application;

11B is a schematic diagram of an exemplary interface when a user triggers an operation instruction in the scenario of FIG. 11A according to an embodiment of the present application;

FIG. 11C is a schematic diagram of an exemplary interface after a user-triggered position in the scene of FIG. 11B is associated with a first image provided by an embodiment of the present application;

12 is a schematic flowchart of another display method provided by an embodiment of the present application;

13A-1 is a schematic diagram of an exemplary scenario of a first screenshot solution provided by an embodiment of the present application;

13A-2 is a schematic diagram of an exemplary scenario of a second screen capture solution provided by an embodiment of the present application;

13A-3 is a schematic diagram of an exemplary scenario of a third screenshot solution provided by an embodiment of the present application;

13A-4 are schematic diagrams of exemplary scenarios of a fourth screen capture solution provided by an embodiment of the present application;

13B is a schematic diagram of an exemplary user interface of a second terminal device in the scenarios of FIGS. 13A-1 to 13A-4 provided by an embodiment of the present application;

13C is a schematic diagram of an exemplary user interface in the scenarios of FIGS. 13A-3 and 13A-4 provided by an embodiment of the present application;

14A is a schematic diagram of a scene in which a first terminal device shoots a video according to an embodiment of the present application;

FIG. 14B is a schematic diagram of an exemplary scene in which the second terminal device according to the embodiment of the application displays the scene captured in FIG. 14A .

Detailed ways

The embodiments of the present application provide a display method, a position determination method, and an apparatus, which are used to solve the problem that the traditional remote assistance function based on AR technology cannot realize the real "hands-on" remote assistance.

The terms "first", "second", "third", "fourth", etc. (if present) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

For ease of understanding, an application scenario of the embodiments of the present application is briefly introduced first. The embodiments of the present application may be applied to a remote assistance scenario, where the remote assistance scenario may be a video call scenario. It can be understood in conjunction with FIG. 1 , it is not difficult to understand that, in an actual application, user A and user B conduct a video call, then the terminal device 110 used by user A displays the image captured by the terminal device 120 used by user B, However, what is displayed on the terminal device 120 is the image captured by the terminal device 110 . In the embodiment of the present application, during the video call between user A and user B, user A or user B can selectively enable the remote assistance function. Taking user A enabling the remote assistance function as an example, user A enables the remote assistance function After that, the terminal device 110 may no longer display the image captured by the terminal device 120 , but display the image captured by the terminal device 110 , and the terminal device 120 may still display the image captured by the terminal device 110 .

Further, if user A has enabled the remote assistance function, after the terminal device 110 sends the captured image 101 to the terminal device 120, the terminal device 120 can display the image, such as the image 101 shown in FIG. perform a corresponding operation on the terminal device 120, for example, click the screen position corresponding to the target object in the image 101, such as the point P shown in FIG. 1, on the screen of the terminal device 120. The terminal device 120 sends the screen position information corresponding to the point P, such as screen coordinates, to the terminal device 110, and the terminal device 110 renders an auxiliary mark on the screen of the terminal device 110 corresponding to the screen position described by the aforementioned screen position information, for example, the terminal device 110 is on the terminal device 110. An auxiliary mark is rendered on the screen of the device 110 at a screen position corresponding to the aforementioned received screen coordinates, so as to prompt the user A to perform a corresponding operation based on the auxiliary mark. Since there is a certain time difference between the time when the terminal device 110 sends the image 101 to the terminal device 120 and the time when the terminal device 120 sends the screen position information corresponding to point P to the terminal device 110, when the terminal device 110 receives the screen position information , the image 102 captured by the terminal device 110 is different from the image 101 , so the screen position information received by the terminal device 110 does not match the current position of the point P in the image 102 . As a result, the position of the auxiliary marker rendered by the terminal device 110 is inconsistent with the position of point P in the image 102 , so that the effect of the remote assistance function is not good, and the real "hands-on" remote assistance cannot be realized.

It should be noted that the above-mentioned video call scenario is only an application scenario of the embodiments of the present application, and the embodiments of the present application can also be applied to other scenarios, such as the scenario of remote home monitoring. The first image of the terminal device 110 is sent to the terminal device 110. If the terminal device 110 enables the remote assistance function, the terminal device 110 can send the first image to the terminal device 120. The user who uses the terminal device 120, such as the aforementioned user B, can respond to the received perform a corresponding operation on the first image of the first image, such as clicking a certain position on the first image, or drawing a circle at a certain position on the first image, etc., and send the screen position information corresponding to the operation to the terminal device 110 , the terminal device 110 may render an auxiliary mark on the screen of the terminal device 110 corresponding to the screen position described by the aforementioned screen position information to prompt the user using the terminal device 110, such as the aforementioned user A, to perform a corresponding operation based on the auxiliary mark, or to prompt the user A The position corresponding to the auxiliary mark is abnormal and so on. However, considering that the camera may shake or change the shooting angle during the shooting process, there is also the aforementioned problem that the effect of hand-to-hand remote assistance cannot be achieved.

Of course, the embodiments of the present application may also be applied to other scenarios, which are not listed and described one by one here.

In order to solve the above problems, the embodiments of the present application provide a display method, a position determination method, and an apparatus, which can solve the above problems. The display method provided by the embodiments of the present application will be described below with reference to the accompanying drawings.

It should be noted that, in the following description of the embodiments of the present application, unless otherwise specified, the foregoing video call scenario is used as an example for description, and the implementation of other scenarios is similar to that of the video call scenario.

Referring to FIG. 2 , which is a schematic flowchart of a display method provided by an embodiment of the present application. The display method provided by the embodiment of the present application may be implemented through the following steps 101-105.

Step 101: The first terminal device sends a first image to the second terminal device, where the first image is an image including a target object.

It should be noted that the first terminal device and the second terminal device mentioned in the embodiments of the present application may be mobile terminal devices such as smart phones and tablet computers, or terminal devices such as desktop computers, which are not described in the embodiments of the present application. Specific restrictions.

It should be noted that the embodiment of the present application does not specifically limit the target object, and the target object may be an object included in the real environment where the first terminal device is located or a certain part of the object, for example, the target object may be a keyboard, A key on a keyboard, a table, a certain position on the table top, a cabinet, a doorknob of a cabinet, a cup and a cup handle, etc.

It should be noted that, in this embodiment of the present application, the first image may be an image captured by the first terminal device, for example, corresponding to the foregoing video call scene. The first image may also be sent by other devices to the first terminal device, for example, corresponding to the foregoing remote home monitoring scenario, the first image is sent to the first terminal device after being captured by the camera.

Step 102: The first terminal device receives first location information from the second terminal device, where the first location information corresponds to the user's operation on the second terminal device for the first location on the target object and corresponds to the location in the first image .

After the second terminal device receives the first image sent by the first terminal device, the second terminal device may display the first image on the screen of the second terminal device. This embodiment of the present application does not specifically limit the first image on the second terminal. With regard to the display mode on the screen of the device, the first image can be displayed in full screen on the screen of the second terminal device, or can be displayed in a specific display area of the second terminal device.

After the first image is displayed on the screen of the second terminal device, the user using the second terminal device can trigger an operation on the second terminal for the first position on the target object, for example, click on the second terminal device and the target object The screen position corresponding to the first position on the target object; another example, draw a circle at the screen position corresponding to the first position on the target object; another example, mark the text at the screen position corresponding to the first position on the target object, etc. . It can be understood that the user triggering the operation on the first position on the target object on the second terminal device may actually be to prompt the user using the first terminal device to pay attention to the first position on the target object, for example, prompting the user to use The user of the first terminal device performs a corresponding operation with respect to the first position on the target object. Therefore, in this embodiment of the present application, the second terminal device may determine that the first position on the target object can be described as being in the the first position information of the image position in the first image, and send the first position information to the first terminal device, so that the first terminal device can further perform corresponding operations according to the first position information to achieve The effect of "hand in hand" remote assistance in a practical sense.

It should be noted that this embodiment of the present application does not specifically limit the first position information, and the first position information may be, for example, information describing the absolute image position of the first position on the target object in the first image, such as the first position information. The position information may be the coordinates describing the absolute image position of the first position on the target object in the first image; the first position information may be, for example, the relative image position describing the first position on the target object in the first image For example, the first position information may be a coordinate describing the relative image position of the first position on the target object in the first image, which is not specifically limited in this embodiment of the present application. Taking into account practical applications, to determine the absolute image position information of the first position on the target object in the first image, the image resolution of the first image and the first position on the target object in the first image can be combined. The relative image position in the image is determined, wherein the image resolution of the first image can be sent by the first terminal device to the second terminal device. In an implementation manner of the embodiment of the present application, in order to reduce the amount of data sent by the first terminal device to the second terminal device, the first position information may be the aforementioned description of the first position on the target object in the first image. Information about the relative image position. In this case, the second terminal device may determine the first position information according to the aforementioned screen position and the position of the display area of the first image on the screen of the second terminal device.

Regarding the method of determining the information describing the relative position of the first position on the target object in the first image, an example will be given now. Assuming that the screen resolution of the second terminal device is 540*480, the screen of the second terminal device may be The construction of the screen coordinate system can be understood by referring to FIG. 3 , which is a schematic diagram of a screen coordinate system provided by an embodiment of the present application. In FIG. 3 , the origin (0,0) of the screen coordinate system of the second terminal device is point O at the lower left corner of the display screen of the second terminal device, and the X coordinate axis of the screen coordinate system passes through the origin O(0,0) And parallel to one side OX of the display screen, the Y coordinate axis of the screen coordinate system passes through the origin O(0,0) and is parallel to one side OY of the display screen. As shown in the figure, the coordinate range of the screen coordinate system is (0,0) to (480,540); if the display area of the first image on the second terminal device is the entire screen, that is, the first image is displayed on the second terminal device , and the coordinates corresponding to the aforementioned screen positions are (480, 270), then the first position information may be coordinates (480/480, 270/540,), ie (1, 0.5). Regarding the case where the first image is displayed in a specific display area on the screen of the second terminal device, the calculation method of the first position information is similar, and will not be described in detail here.

Step 103: The first terminal device determines, according to the first position information, that the first position on the target object corresponds to the second position in the virtual coordinate system, and adds a mark at the second position in the virtual coordinate system; the virtual coordinate system The second position in , describes the actual position of the first position on the target object in the real three-dimensional space.

It should be noted that the virtual coordinate system may be constructed by the first terminal device. Specifically, after the user using the first terminal device starts the remote assistance function on the first terminal device, the first terminal device may, for example, start the AR service. , the AR service can construct the virtual coordinate system and construct virtual objects corresponding to objects in the real three-dimensional space. The coordinates in the virtual coordinate system of the virtual object corresponding to the object in the real three-dimensional space can reflect the actual position of the object in the real three-dimensional space. Specifically, for a video call scene, the AR service can, for example, obtain multiple frames of images continuously shot by the first terminal device, and analyze some or all of the feature points on the surface of the object in the three-dimensional space, so as to determine the corresponding feature of the object. The coordinates in the virtual coordinate system of the virtual object. For the aforementioned remote home monitoring scenario, the AR service can, for example, acquire multiple frames of images sent by the camera to the first terminal device, and analyze some or all of the feature points on the surface of the object in the three-dimensional space where the target object is located, so as to determine The coordinates in the virtual coordinate system of the virtual object corresponding to the object in the three-dimensional space where the target object is located are obtained. The aforementioned objects may be some feature points, such as points corresponding to the four corners of the keyboard, or may be a virtual plane obtained by fitting according to the feature points, such as a virtual plane corresponding to the ground in a real three-dimensional space. Another example is the AR service that can obtain the depth information of the object according to the depth camera and radar and other equipment, so as to fit the virtual plane according to the depth information. For example, for the express box in the real three-dimensional space, the corresponding express can be fitted according to the depth information. virtual planes of multiple surfaces of the box, thereby determining the coordinates of the virtual object corresponding to the object in the virtual coordinate system.

In an implementation manner of this embodiment of the present application, for a video call scenario, the origin of the virtual coordinate system constructed by the first terminal device can be used when the first terminal device starts the AR service, and the first terminal device is in a real three-dimensional The position in the stereo space is determined, for example, the origin of the virtual coordinate system corresponds to the position of the first terminal device in the real three-dimensional stereo space, that is, the coordinates of the first terminal device in the world coordinate system correspond to the coordinates of the origin in the virtual coordinate system. For the aforementioned remote home monitoring scenario, considering that the camera and the first terminal device may not be in the same three-dimensional space, for example, the camera is in the living room, and the first terminal device is in the bedroom or even the office, etc., for this situation , the origin of the virtual coordinate system can correspond to the position of the camera in the real three-dimensional space, that is, the coordinates of the camera in the world coordinate system correspond to the coordinates of the origin in the virtual coordinate system. In an implementation manner of the embodiment of the present application, the virtual coordinate system includes three coordinate axes, which are an X-axis, a Y-axis, and a Z-axis, respectively. The Y axis may be the opposite direction of gravity, the X axis may be, for example, the projection direction of the usb port on the first terminal device on the horizontal plane, and the Z axis is the direction perpendicular to both the X axis and the Y axis.

It can be understood that, for a video call scenario, if the first terminal device is a mobile device such as a smartphone, each time the first terminal device starts the AR service, the virtual coordinate origin and the three coordinate axis directions set by the AR service are all the same. It is determined by the state when the first terminal device starts the AR service, such as the position of the first terminal device in the real three-dimensional space, and the posture of the first terminal device itself. Therefore, each time the first terminal device starts the AR service The origin of the established virtual coordinate system and the three coordinate axes may not be consistent.

Of course, the above is only an exemplary description. The origin of the virtual coordinate system and the direction of the three coordinate axes can also be determined in other ways, which are related to the settings of the AR service itself, and will not be listed here. . It should be noted that the AR service mentioned in the embodiments of this application may be, for example, an augmented reality software development kit (AR SDK) in traditional technologies, such as AREngine provided by Huawei and ARCore provided by Google. , ARKit provided by Apple, etc. Of course, the AR service may also be redeveloped, which is not specifically limited in this embodiment of the present application.

In this embodiment of the present application, a mark may be added at a first position on the target object corresponding to a second position in the virtual coordinate system by using the aforementioned AR service. The embodiment of the present application does not specifically limit the mark, and the mark may be, for example, a five-pointed star, an arrow, or the like. Considering that some markers themselves have directions, such as arrow directions, different directions have different display modes, so in an implementation manner of the embodiment of the present application, when the first terminal device adds the marker, it can also obtain the virtual coordinates of the marker. The direction in the system, so as to add a mark corresponding to the direction in the virtual coordinate system, such as adding an arrow pointing downward.

Step 104: The first terminal device renders the aforementioned mark on the second image, so that the corresponding position of the mark on the second image is consistent with the position of the first position on the target object in the second image; the second image is the first An image displayed by a terminal device including the target object.

It should be noted that, in the scenario of remote assistance, the second image may be an image captured at a certain moment after the first terminal device captures the first image and displayed on the first terminal device, for example, the first image is at time k The captured image, the second image is an image captured at time k+m and displayed on the first terminal device. In other scenarios, such as in the aforementioned remote home monitoring scenario, the second image may be an image captured at a certain moment after the camera captures the first image, sent to the first terminal device, and displayed by the first terminal device.

In the embodiment of the present application, for the aforementioned video call scenario, it is considered that even if the position of the first terminal device changes after sending the first image to the second terminal device, the target corresponding to the operation triggered by the user on the second terminal The actual position of the first position on the object in the real three-dimensional space usually does not change within the above-mentioned time difference. Similarly, for the aforementioned remote home monitoring scenario, even if the camera shakes or the shooting angle changes after capturing the first image, the first position on the target object corresponding to the operation triggered by the user on the second terminal is in the real three-dimensional space. The actual position in the stereoscopic space usually does not change within the above time difference. In this embodiment of the present application, the first position on the target object determined by the first terminal device corresponds to the second position in the virtual coordinate system, which can reflect the actual position of the first position on the target object in the real three-dimensional space. Location. After it is determined that the first position on the target object corresponds to the second position in the virtual coordinate system, a mark can be added at the second position in the virtual coordinate system, and the first terminal device that includes the target object is displayed. The mark is rendered on the second image, so that the corresponding position of the mark on the second image is consistent with the position of the first position on the target object in the second image, so as to achieve the effect of "hand in hand" remote assistance.

The embodiments of the present application do not specifically limit the specific implementation manner of rendering the mark on the second image. As an example, a rendering engine provided by the AR service may be invoked to render the mark on the second image. Specifically, the projection matrix of the first terminal device and the field of view information corresponding to the second image can be provided to the rendering engine, and the rendering engine can use the projection matrix of the first terminal device, the field of view information corresponding to the second image, and the mark in the virtual The second position in the coordinate system, render the mark on the screen of the first terminal device, so that the position corresponding to the mark on the second image is consistent with the position of the first position on the target object in the second image .

This embodiment of the present application does not specifically limit the field of view information corresponding to the second image, and the field of view information may be used to describe the position of the camera that captures the second image in the aforementioned virtual coordinate system when the second image is captured, and the position of the camera that captures the second image. When taking an image, the shooting direction of the camera that shoots the second image in the virtual coordinate system. In this embodiment of the present application, the first terminal device may obtain the field of view information corresponding to the second image by using the aforementioned AR service.

Step 105: The first terminal device displays the mark rendered on the second image.

It can be understood that, after the first terminal device renders the mark on the second image, to a certain extent, it can indicate that the mark and the first position on the target object can be displayed in the same position on the second image. document. In this embodiment of the present application, after rendering the mark on the second image, the rendered mark may be displayed through a corresponding hardware device, so that the user can view the mark through the content displayed by the hardware device. In this embodiment of the present application, the first terminal device may display the mark, so that the user using the first terminal device can determine, according to the mark, that the user using the second terminal device performs the first operation on the target object corresponding to the foregoing operation. The location, furthermore, enables the user of the first terminal device to perform further operations based on the mark.

It can be seen from the above description that by using the solutions of the embodiments of the present application, the effect of "hand-in-hand" remote assistance can be achieved.

It can be understood that during the video call, the first terminal device will send the multiple frames of images it shoots to the second terminal device. Similarly, for the aforementioned remote home monitoring scenario, the first terminal device will send data from the camera. The multiple frames of images are sent to the second terminal device. In order to determine which frame of image corresponds to when the user triggers an operation on the second terminal device, in this embodiment of the present application, when sending the first image to the second terminal device, the first terminal device may also send the first image to the second terminal device. The identifier of the image is sent to the second terminal device. Correspondingly, when the second terminal device sends the aforementioned first location information to the first terminal device, the identifier of the first image can also be sent to the first terminal device correspondingly, so that the first terminal The device determines that the first location information corresponds to the first image.

The embodiment of the present application does not specifically limit the identification of the first image, for example, the identification of the first image may be the time when the first image was taken; the first identification may be the visual field information corresponding to the first image, and the visual field information is used for When capturing the first image, the target position of the camera that captures the first image in the virtual coordinate system, and the target shooting direction of the camera that captures the first image in the virtual coordinate system when capturing the first image. In this embodiment of the present application, the terminal device may obtain the identifier of the first image by using the aforementioned AR service, so as to send the identifier of the first image to the second terminal device.

The following describes a specific implementation manner of determining that the first position on the target object corresponds to the second position in the virtual coordinate system by the first terminal device according to the first image position in the foregoing step 103 with reference to the accompanying drawings.

Referring to FIG. 4 , which is a schematic flowchart of a method for determining that a first position on a target object corresponds to a second position in a virtual coordinate system according to an embodiment of the present application. The method can be implemented, for example, through the following steps 201-202.

Step 201: The first terminal device determines the image position of the first position on the target object in the image coordinate system corresponding to the first image according to the first position information and the image resolution of the first image.

In this embodiment of the present application, for the aforementioned video call scenario, the first terminal device can obtain the image resolution of the first image through the AR service, and when the first terminal device enables the remote assistance function, the AR service can configure the first terminal device to The image resolution of the captured image. Specifically, the AR service may acquire the screen resolution of the first terminal device, so as to configure an image captured by the first terminal device, for example, the image resolution of the first image, according to the screen resolution of the first terminal device. Generally speaking, the AR service will configure the image resolution of the first image to be smaller than the screen resolution of the first terminal device. For the foregoing remote home monitoring scenario, the image resolution of the first image may be sent to the first terminal device by the camera that captured the first image.

In the case where the aforementioned first position information may be information describing the relative image position of the first position on the target object in the first image, when step 201 is implemented, for example, the first position information and the image of the first image may be combined. The product of the resolution is determined as the image position of the first position on the target object in the image coordinate system corresponding to the first image. For example, if the first position information is the coordinates (0.5, 1.0), and the image resolution of the first image is 480*480, the image position of the first position on the target object in the image coordinate system corresponding to the first image is determined as (480*0.5,480*1.0), which is (240,480).

Step 202: The first terminal device determines, according to the aforementioned image position, that the first position on the target object corresponds to the second position in the virtual coordinate system.

Before introducing the specific implementation of step 202 , the shooting principle of the camera of the first terminal device is first introduced with reference to FIG. 5 . FIG. 5 is a schematic diagram of a visual frustum provided by an embodiment of the present application.

It should be noted that the shape of the viewing frustum determines how to map the image captured by the camera to the screen of the first terminal device. Referring to FIG. 5, point O in FIG. 5 is the position of the camera in the virtual coordinate system, and 502 is the angle of view of the camera; the plane 502 in the virtual coordinate system shown in FIG. 5 corresponds to the screen of the first terminal device, and the plane 503 corresponds to the screen of the first terminal device. The farthest range the camera can shoot. The aforementioned first position on the target object corresponds to the mark added at the second position in the virtual coordinate system, for example, it may be the virtual object 504 shown in FIG. 5 .

When step 202 is specifically implemented, it can be combined with the viewing frustum when the first terminal device captures the first image, first determine the direction in the virtual coordinate system corresponding to the first position on the target object, and then according to the direction and the corresponding direction of the target object. For the virtual object, it is determined that the first position on the target object corresponds to the second position in the virtual coordinate system. Regarding the virtual object corresponding to the target object, the construction of the virtual coordinate system has been described in detail above, and will not be described in detail here.

A possible implementation manner of step 202 is introduced below with reference to FIG. 6 . FIG. 6 is a schematic flowchart of a method for determining that a first position on a target object corresponds to a second position in a virtual coordinate system according to an embodiment of the present application. . The method can be implemented, for example, by the following steps 301-303.

Step 301: The first terminal device acquires field of view information corresponding to the first image.

It should be noted that the field of view information corresponding to the first image can describe the position of the camera that captures the first image in the virtual coordinate system when the first image is captured, and the position of the camera that captures the first image when the first terminal device captures the first image. The shooting direction of the camera of an image in the virtual coordinate system. For a video call scenario, the field of view information corresponding to the first image can describe the position of the camera of the first terminal device in the virtual coordinate system when the first terminal device captures the first image, and the location of the camera of the first terminal device in the virtual coordinate system when the first terminal device captures the first image The shooting direction of the camera of the first terminal device in the virtual coordinate system when an image is displayed. For the convenience of description, "when the first image is captured, the position of the camera that captures the first image in the virtual coordinate system" is referred to as the "target position", and "when the first image is captured, the camera that captures the first image is located in the virtual coordinate system". The shooting direction in the virtual coordinate system" is called the "target shooting direction".

As described above, in this embodiment of the present application, the field of view information corresponding to the first image may be acquired through the AR service.

Step 302: The first terminal device calculates the position of the target ray in the virtual coordinate system according to the image position and the field of view information corresponding to the first image; the target ray is the ray whose endpoint is the target position and passes through the second position in the virtual coordinate system. .

The embodiment of the present application does not specifically limit the specific implementation of calculating the position of the target ray in the virtual coordinate system. As an example, the calculation of the target ray in the virtual coordinate system can be performed by calculating the target equation of the target ray in the virtual coordinate system. s position.

In the embodiment of the present application, the target equation can reflect the direction in the virtual coordinate system corresponding to the first position on the target object, that is, can reflect the direction of the second position in the virtual coordinate system. In an implementation manner of the embodiment of the present application, the target equation may be determined by the following formula (1) and formula (2).

[End]＝[View]^(-1)×[Projection]^(-1)×[p] Formula (1)

Direction=End-Start Formula (2)

In formula (1), [End] is the position of a point on the target ray, and [End] can be a 4*1 column vector, which is a homogeneous expression of coordinate values. For example, [End] can be (XW, YW,ZW,1)-1;

[View]^(-1) represents the inverse matrix of the field of view matrix [View] corresponding to the first image, which is determined by the field of view corresponding to the first image, that is, when the first image is captured, the camera that captures the first image is in the virtual coordinate system and when the first image is captured, the position of the camera that captures the first image in the virtual coordinate system is determined;

It should be noted that [View]^(-1) may also be referred to as the pose matrix when the camera captures the first image, and the pose matrix may be, for example, a (4*4) matrix. The pose matrix when the camera captures the first image, for example, can be [(R_C&T_C@0&1)], where R_C is the rotation matrix of the camera in the world coordinate system, and the rotation matrix can be, for example, a (3*3) matrix; T_C is the translation vector of the camera in the world coordinate system, and T_C can be, for example, a (3*1) column vector. After the first terminal device starts the AR service, the first terminal device can obtain the rotation matrix R_C of the camera in the world coordinate system and the translation vector T_C of the camera in the world coordinate system through the AR service, so as to obtain the pose matrix [View] ^(-1);

[Projection]^(-1) is the inverse matrix of the projection matrix corresponding to the first image. The first terminal device can use the AR service to obtain the projection matrix corresponding to the first image. The parameters determine the projection matrix corresponding to the first image; wherein, the parameters when the camera shoots the first image may include, for example, the horizontal axis focal length fx of the camera, the vertical axis focal length fy of the camera, and the corresponding principal point offset values cx and cy of the camera, The pixel width value width and the pixel height value height of the first image captured by the camera, and so on. Among them, the horizontal axis focal length fx, vertical axis focal length fy of the camera, and the corresponding principal point offset values cx and cy of the camera are related to the hardware configuration of the camera. Generally speaking, for multiple cameras of the same model, the corresponding horizontal axis focal length fx , the vertical axis focal length fy, and the principal point offset values cx and cy are the same;

[p] is the image position of the target object in the first image, similar to [End], [p] can be a 4*1 column vector, which is a homogeneous expression of coordinate values;

Direction represents the target equation;

End is the coordinate representation of [End], indicating the coordinates of a point on the target ray in the virtual coordinate system;

Start represents the target position of the camera that captures the first image in the virtual coordinate system when the first image is captured, that is, the coordinates of the camera that captures the first image in the virtual coordinate system when capturing the first image. In another implementation manner of the embodiment of the present application, the target equation may be determined by the following formula (3) and formula (4).

[End]=[external parameter matrix]×[internal parameter matrix]^(-1)×[p] Formula (3)

Direction=End-Start Formula (4)

In formula (3), [End] is a point position on the target ray;

The extrinsic parameter matrix is determined by the position and direction of the camera in the virtual coordinate system when the first image is captured. The extrinsic parameter matrix can be the pose matrix described above, which is the inverse matrix of the field of view matrix [View] corresponding to the first image. For the description of the extrinsic parameter matrix, reference may be made to the foregoing description of the field of view matrix [View] corresponding to the first image, which will not be described in detail here.

The internal parameter matrix is determined according to the internal parameters of the camera corresponding to the first image, for example, can be determined according to the focal length fx of the horizontal axis of the camera, the focal length of the vertical axis fy of the camera, and the principal point offset values cx and cy corresponding to the camera. As mentioned above, the focal length of the horizontal axis fx, the focal length of the vertical axis fy, and the corresponding principal point offset values cx and cy of the camera are related to the hardware configuration of the camera, so the internal parameter matrix can be determined by obtaining the identification of the camera, such as the model of the camera. .

The formula (4) is the same as the formula (2), and is not repeated here.

In this embodiment of the present application, the target equation can also be calculated in other ways. For example, instead of using a matrix for calculation, it can be directly calculated from the internal parameters of the camera corresponding to the first image and the position of the camera in the virtual coordinate system when the first image is captured. and direction calculation target equation, for example, considering that the matrix itself can be divided into multiple equations, therefore, in this embodiment of the present application, it is also possible to directly use the "expression expressed by the matrix" equivalent to " The expression expressed by the equation", to calculate the aforementioned target equation. However, the basic principle is the same whether you use matrices for calculations or equations for calculations.

Step 303: The first terminal device determines the intersection position of the target ray and the virtual object in the virtual coordinate system as the first position on the target object corresponding to the second position in the virtual coordinate system.

It can be understood that, for the camera, it can shoot objects in the real three-dimensional space into a two-dimensional image. Therefore, for a ray such as all points on the target ray, it is reflected as a two-dimensional image on the two-dimensional image. point. Therefore, after determining the equation of the target ray in the virtual coordinate system, it is also possible to combine the virtual object in the virtual coordinate system to determine that the first position on the target object corresponds to the second position in the virtual coordinate system.

The intersection of the target ray and the virtual object actually corresponds to the first position on the target object corresponding to the screen position where the user triggers the operation on the second terminal device and corresponds to the second position in the virtual coordinate system.

In this embodiment of the present application, the collision function provided by the AR service can be called to calculate the intersection position of the target ray and the virtual object in the virtual coordinate system, so as to determine that the first position on the target object corresponds to the second position in the virtual coordinate system . The embodiment of the present application does not specifically limit the collision function, and the collision function may be, for example, the hitTest function provided by ARCore.

The display method applied to the first terminal device according to the embodiments of the present application has been described above, and a method for determining a position applied to the second terminal device is described below with reference to the accompanying drawings.

It should be noted that the position determination method applied to the second terminal device is similar to the basic principle of the aforementioned display method applied to the first terminal device. After an image is sent to the second terminal device, the position changes, but the actual position of the first position on the target object corresponding to the operation triggered by the user on the second terminal within the above-mentioned time difference usually does not occur in the real three-dimensional space. changed. Similarly, for the aforementioned remote home monitoring scenario, even if the camera shakes or the shooting angle changes after capturing the first image, the first position on the target object corresponding to the operation triggered by the user on the second terminal is in the real three-dimensional space. The actual position in the stereoscopic space usually does not change within the above time difference. Therefore, in this embodiment of the present application, the second terminal device may determine, according to the aforementioned first position information, that the first position on the target object corresponds to the second position in the virtual coordinate system, and the first position on the target object corresponds to The second position in the virtual coordinate system describes the actual position of the first position on the target object in the real three-dimensional space. After determining that the first position on the target object corresponds to the second position in the virtual coordinate system, the first position on the target object corresponding to the second position in the virtual coordinate system may be sent to the first terminal device, so as to facilitate The first terminal device adds a marker at the second position in the virtual coordinate system, and renders the marker on the second image including the target object displayed on the first terminal device, and displays the marker rendered on the second image. mark.

Referring to FIG. 7 , which is a schematic flowchart of a method for determining a location provided by an embodiment of the present application. The method can be implemented, for example, through the following steps 401-403.

Step 401: The second terminal device receives the first image sent by the first terminal device, and displays the first image on the screen of the second terminal device; the first image is an image including the target object.

Step 402: The second terminal device receives an operation instruction triggered by the user on the second terminal device for the first position on the target object, and determines the first position information, the first position information corresponds to the user on the second terminal device The operation on the first position on the target object corresponds to the position in the first image.

In an implementation manner of the embodiment of the present application, the first position information may be information describing a relative image position of the first position on the target object in the first image. In this case, the second terminal device may determine the first position information according to the aforementioned screen position and the display area of the first image on the screen of the second terminal device.

It should be noted that the principles of steps 401-402 are similar to the aforementioned steps 101-102, except that the aforementioned steps 101-102 are performed by the first terminal device, while steps 401-402 are performed by the second terminal side. Therefore, For the description of steps 401-402, reference may be made to the description of steps 101-102 above, which will not be described in detail here.

Step 403: The second terminal device determines, according to the first position information, that the first position on the target object corresponds to the second position in the virtual coordinate system, and corresponds the first position on the target object to the second position in the virtual coordinate system Sent to the first terminal device; the second position in the virtual coordinate system reflects the actual position of the first position on the target object in the real three-dimensional space.

For the description of the virtual coordinate system, reference may be made to the relevant description part in step 103 above, which will not be described in detail here.

It should be noted that the implementation manner in which the second terminal device determines that the first position on the target object corresponds to the second position in the virtual coordinate system according to the first position information is the same as the first terminal device determining the target object according to the first position information. The first position of the target object corresponds to the second position in the virtual coordinate system, both of which are based on the aforementioned first position information and the image resolution of the first image to determine the first position on the target object in the image coordinate system corresponding to the first image. Then, the second position in the virtual coordinate system corresponding to the first position on the target object is determined according to the image position of the first position on the target object in the image coordinate system corresponding to the first image.

It is only considered that when the first terminal device sends the first image to the second terminal device, in order to reduce the bandwidth occupied by sending the first image, the compressed image may be sent; The first image itself may not be able to obtain the image resolution of the first image. Therefore, when the first terminal device sends the first image to the second terminal device, it can also send the image resolution of the first image to the second terminal. device, so that the second terminal device can determine the image position of the first position on the target object in the image coordinate system corresponding to the first image according to the first position information and the image resolution of the first image, and further, according to the target object The first position on the target object corresponds to the image position in the image coordinate system corresponding to the first image, and it is determined that the first position on the target object corresponds to the second position in the virtual coordinate system.

It should be noted that, according to the image position of the first position on the target object in the image coordinate system corresponding to the first image, the second terminal device determines that the first position on the target object corresponds to the second position in the virtual coordinate system. The specific implementation method is similar to the first terminal device determining the first position on the target object corresponding to the second position in the virtual coordinate system according to the image position of the first position on the target object in the image coordinate system corresponding to the first image. The specific implementation methods are the same, both are based on the visual field information corresponding to the first image and the image position of the first position on the target object in the image coordinate system corresponding to the first image, to calculate the position of the target ray in the virtual coordinate system; is the ray whose endpoint is the target position and passes through the second position in the virtual coordinate system; then the intersection position of the target ray and the virtual object in the virtual coordinate system is determined as the first position on the target object corresponding to the virtual coordinate system the second position.

It is just considering that the first image is not an image captured by the second terminal device, so the field of view information corresponding to the first image may not be obtained by the second terminal device. Therefore, the first terminal device sends the first image to the second terminal. device, the field of view information corresponding to the first image can also be sent to the second terminal device, so that the second terminal device can determine that the first position on the target object corresponds to the virtual coordinate system according to the field of view information corresponding to the first image the second position.

It can be understood that, when the second terminal device determines that the first position on the target object corresponds to the second position in the virtual coordinate system according to the field of view information corresponding to the first image, it needs to combine the relevant information of the virtual coordinate system, such as The position of the virtual object corresponding to the target object in the virtual coordinate system. According to the foregoing description of the virtual coordinate system, the virtual coordinate system is constructed by the first terminal device according to the information in the real three-dimensional space where the target object is located, such as the position of the object in the aforementioned real three-dimensional space. Therefore, if the second terminal device and the target object are not in the same environment, the second terminal device itself cannot construct the same virtual coordinate system constructed by the first terminal device according to the information in the real three-dimensional space where the target object is located. In this case, the first terminal device can send the information related to the virtual coordinate system constructed by the first terminal device to the second terminal device, for example, the virtual object corresponding to the target The position in the constructed virtual coordinate system is sent to the second terminal device. After receiving the information related to the virtual coordinate system constructed by the first terminal device, the second terminal device can determine that the first position on the target object corresponds to the virtual coordinate system constructed by the first terminal device in combination with the information related to the virtual coordinate system constructed by the first terminal device. The second position in the coordinate system.

After the second terminal device sends the first position on the target object corresponding to the second position in the virtual coordinate system to the first terminal device, the first terminal device may add a mark at the second position in the virtual coordinate system, and render the mark on the second image, and display the mark rendered on the second image; the second image is the image including the target object displayed by the first terminal device; the first position on the target object corresponds to the position in the virtual coordinate system. The second position reflects the actual position of the first position on the target object in the real three-dimensional space.

After the second terminal device sends the first position on the target object corresponding to the second position in the virtual coordinate system to the first terminal device, the first terminal device adds a mark at the second position in the virtual coordinate system, and adds a mark at the second position in the virtual coordinate system. For an implementation manner of rendering the mark on the second image and displaying the mark rendered on the second image, reference may be made to the relevant description parts of steps 103-105 in the foregoing embodiment, which will not be described in detail here.

It should be noted that this embodiment of the present application does not specifically limit the specific information sent by the second terminal device to the first terminal device, as long as the first terminal device can determine, according to the information, that the first position on the target object corresponds to the virtual coordinate system. the second position. In view of this, in an implementation manner of this embodiment of the present application, step 403 may be replaced by the second terminal device may determine the position of the aforementioned target ray in the virtual coordinate system according to the first position information, and place the target ray in the virtual coordinate system The position in the system is sent to the first terminal device. Thus, the first terminal device calculates the first position on the target object corresponding to the second position in the virtual coordinate system according to the position of the target ray in the virtual coordinate system, and further, the first position of the first terminal device on the target object A marker is added correspondingly at a second location in the virtual coordinate system, the marker is rendered on the second image, and the rendered marker on the second image is displayed.

In the above embodiment, the video stream sent by the first terminal device to the second terminal device includes a plurality of video frames, and each video frame is an image. Correspondingly, after the first terminal device sends the first image to the second terminal device, it still continues to send the image to the second terminal device. Then, from the moment when the second terminal device starts to receive the image of the first terminal device to the moment when the first image is received, the second terminal device has continuously received multiple images. The user on the second terminal device triggers the operation instruction when seeing the first image. After the user actually starts to input the operation instruction for the first position on the target object on the second terminal device, what he sees may not be the first image. Rather, the first terminal device sends other images (eg, the ith image) after sending the first image. The user-triggered operation instruction on the target object will be associated with the first image, that is, the location information of the first position will be associated with the first image, and the first terminal device may occur within the time period when the user inputs the operation instruction jitter, which causes the screen that the user sees to change during the process of inputting operation instructions on the second terminal device, so that the user on the second terminal device may have the illusion that the input position is incorrect, thus making the effect of the remote assistance function ineffective. it is good.

Exemplarily, as shown in FIG. 11A , the first terminal device continues to send images to the second terminal device, and accordingly, the second terminal device receives image 1101 , image 1102 , image 1103 , image 1104 and image 1105 in sequence. The image 1101 is, for example, the first image described in the above embodiment. When the second terminal device receives and displays the image 1101, the user on the second terminal device side triggers the second terminal device to input an operation instruction. At the moment when the operation instruction is received, for example, the image 1105 is displayed by the second terminal device. As shown in FIG. 11B , the position where the user starts triggering on the second terminal device is, for example, the position P of the spherical object, and when the user completes the triggering, an image 1105 is displayed (as shown in FIG. 11C ). However, the second terminal device receives the operation instruction triggered by the user, and associates the position triggered by the user with the image 1101 . If the first terminal device shakes during the sending of the image 1101 to the image 1105, after the triggering by the user is completed, the position P on the image 1101 is displayed as the position Q shown in FIG. 11C due to the screen change. As shown in FIG. 11C , the visual effect given to the user by the position Q is not the spherical object that the user wants to mark. Therefore, the user may think that the position triggered by himself is wrong. It can be seen that the above embodiment may make the user experience on the second terminal device side not good.

In order to solve this problem, an embodiment of the present application also provides a display method. As shown in FIG. 12 , the method can be implemented, for example, through the following steps 501 to 506 .

Step 501: The second terminal device acquires a first image, and statically displays the first image on the screen of the second terminal device; the first image is an image including a target object.

Wherein, "static display" means that after the second terminal device acquires the first image, the first image is displayed on the main interface of the second terminal device, even if the second terminal device still continuously receives images from the first terminal device, all the Other received images are not displayed on the main interface, for example, they can be displayed through a small window on the main interface, or other received images may not be displayed.

Understandably, when performing remote assistance, the first terminal device, as the party for assistance, displays the video collected by the first terminal device, and the second terminal device, as the party to assist, displays the video collected and sent by the first terminal device. .

The trigger timing of the screenshot may be triggered when the remote assistance function is enabled, and the enablement of the remote assistance function may be triggered by a user on the side of the first terminal device or a user on the side of the second terminal device. In an implementation manner, when the user on the side of the first terminal device triggers the first terminal device to send a remote assistance request to the second terminal device, the user on the side of the first terminal device takes a screenshot of the video on the side of the first terminal device to obtain the first image. Further, the first terminal device sends the first image to the second terminal device. In the second implementation manner, the user on the side of the first terminal device triggers the first terminal device to send a remote assistance request to the second terminal device, and when the second terminal device receives the remote assistance request, the second terminal device sends a request to the second terminal device. Take a screenshot of the video shown above to get the first image. In a third implementation manner, the user on the side of the second terminal device triggers the remote assistance function to be enabled, the second terminal device sends a video screenshot request to the first terminal device, and the first terminal device receives the video screenshot request and sends the first terminal device to the first terminal device. Take a screenshot of the video displayed above to obtain the first image and send it to the second terminal device.

The trigger timing of the screenshot may be triggered by a user on the side of the first terminal device or a user on the side of the second terminal device after the remote assistance function has been enabled. In an implementation manner, during the remote assistance process, the user on the side of the first terminal device triggers the "screenshot" control on the interface of the first terminal device, the first terminal device receives the screenshot instruction, and further, the user on the side of the first terminal device triggers the "screenshot" control. Take a screenshot of the video to get the first image. After that, the first terminal device sends the first image to the second terminal device. In another implementation manner, during the remote assistance process, the user on the side of the second terminal device triggers the "screenshot" control on the interface of the second terminal device, the second terminal device receives the screenshot instruction, and then the second terminal device Take a screenshot of the video on the side to get the first image. In another implementation manner, during the remote assistance process, the user on the side of the second terminal device triggers the "screenshot" control on the interface of the second terminal device, the second terminal device receives the screenshot instruction, and then sends a message to the first terminal device. The first terminal device receives the video screenshot request, and takes a screenshot of the video displayed on the first terminal device, so as to obtain the first image and send it to the second terminal device.

Exemplarily, as shown in FIG. 13A-1, the second terminal device can continuously receive images from the first terminal device, for example, the second terminal device receives image 1301, image 1302, image 1303, image 1304, and image 1305 in sequence . After sending the image 1301 to the second terminal device, the first terminal device sends, for example, a remote assistance request to the second terminal device in response to a trigger from the user on the side of the first terminal device. In this embodiment, after receiving the trigger from the user, the first terminal device takes a screenshot to obtain the image 1302, and then sends the image 1302 together with the remote assistance request to the second terminal device. The second terminal device receives the image 1302 and displays the image 1302 on the main interface. As shown in FIG. 13B , the second terminal device still continues to receive the image 1303 , the image 1304 and the image 1305 , and displays the image 1303 , the image 1304 and the image 1305 through the small window 131 on the main interface where the image 1302 is displayed, and the received subsequent images. In addition, another possible implementation manner is that after receiving the trigger from the user, the first terminal device first sends a remote assistance request to the second terminal device, and then sends the first image to the second terminal device. No limit here.

As shown in FIG. 13A-2, the first terminal device sends an image 1301, an image 1302, an image 1303, an image 1304, and an image 1305 to the second terminal device in sequence. After sending the image 1301 to the second terminal device, the first terminal device sends, for example, a remote assistance request to the second terminal device in response to a trigger from the user on the side of the first terminal device. In this embodiment, after receiving the remote assistance request, the second terminal device takes a screenshot on the second terminal device to obtain the image 1302 . Further, as shown in FIG. 13B , the second terminal device displays the image 1302 on the main interface, continues to receive the image 1303 , the image 1304 and the image 1305 , and displays the image 1303 through the small window 131 on the main interface where the image 1302 is displayed. , Image 1304 and Image 1305, and subsequent images received.

As shown in FIG. 13A-3, the first terminal device has initiated a remote assistance request to the second terminal device. The first terminal device sends an image 1301, an image 1302, an image 1303, an image 1304, and an image 1305 to the second terminal device in sequence. After the first terminal device sends the image 1301 to the second terminal device, for example, a screen capture request input by the user on the side of the first terminal device is received. In response to the screenshot request, the first terminal device takes a screenshot to obtain the image 1302, and then sends the image 1302 to the second terminal device. The manner in which the second terminal device displays the image 1302 and the image 1303 , the image 1304 and the image 1305 is shown in FIG. 13B , which will not be described in detail here.

As shown in Figures 13A-4, the first terminal device has initiated a remote assistance request to the second terminal device. The first terminal device sends an image 1301, an image 1302, an image 1303, an image 1304, and an image 1305 to the second terminal device in sequence. After receiving the image 1301, the second terminal device, for example, receives a screen capture request input by the user. In response to the screenshot request, the second terminal device takes screenshots to obtain the image 1302 . The manner in which the second terminal device displays the image 1302 and the image 1303, the image 1304 and the image 1305 is shown in Fig. 13B, and will not be described in detail here.

It should be pointed out that, in some embodiments, in the embodiments illustrated in FIGS. 13A-3 and 13A-4 , the user can trigger the “screenshot” shortcut key of the corresponding terminal device, so that the corresponding terminal device performs a screen capture operation. In other embodiments, during the execution of the remote assistance function, at least one of the first terminal device and the second terminal device may display a "screenshot" control on the interface. Exemplarily, as shown in FIG. 13C , the "screen capture" control displayed on the interface of the terminal device may be a button 132 marked with a "screen capture" logo. Of course, in other embodiments, the "screen capture" control displayed on the interface of the terminal device may be an icon indicating "screen capture". No limit here.

It can be understood that FIGS. 13A-1 to 13A-4 are only schematic descriptions, and do not limit the method for acquiring the first image in this embodiment. In other implementation manners, the first terminal device may also respond to an instruction of the second terminal device, take a screenshot to obtain the first image, and then send the first image to the second terminal device. Not detailed here.

Understandably, when the remote assistance function is disabled, or the user triggers to cancel the screenshot, the second terminal device continues to display the video stream sent by the first terminal device. Exemplarily, the interface illustrated in FIG. 13B may further include, for example, a button 133 for triggering the closing of the remote assistance function, and the button 133 may be marked, for example, with "complete assistance". Furthermore, after the user on the second terminal device side triggers the mark at the position P of the image 1302 , the remote assistance function can be turned off by clicking the button 133 . Furthermore, after receiving the close instruction, the second terminal device can enlarge the small window 131 to the entire screen to display the video stream sent by the first terminal device on the main interface of the second terminal device.

Before the screenshot is triggered, the video stream from the first terminal device is displayed on the second terminal device, that is, it changes with the change of the video content collected and sent by the first terminal device. After the screenshot is triggered, the first image is displayed on the second terminal device instead of the video stream, and the second terminal device can still receive the video stream sent by the first terminal device but does not display it; What is displayed on the main interface is the first image instead of the video stream, and the video stream is displayed in a small window on the screen of the second terminal device (as shown in the embodiment illustrated in FIG. 13B ). After the screenshot is triggered, the first terminal device can continue to send the video stream to the second terminal device, or it can stop sending the video stream until the user triggers to cancel the screenshot and then resume sending the video stream, or stop sending the video stream until the user on the first terminal device side. Or the user on the second terminal device side triggers the completion of the remote assistance and then resumes sending the video stream.

Step 502: The second terminal device receives an operation instruction triggered by the user on the first image for the first position on the target object, and determines the first position information.

Wherein, after the screenshot is taken, the main interface of the second terminal device always displays the first image, and during the process of triggering the mark by the user on the side of the second terminal device, the screen will not change due to the shaking of the first terminal device. The first position of the target object seen and triggered by the user is the first position of the target object displayed on the first image. The first position information corresponds to an operation performed by the user on the second terminal device with respect to the first position on the target object and corresponds to a position in the first image. In this way, the user experience on the second terminal device side can be improved.

Exemplarily, the image 1302 is always displayed on the main interface of the second terminal device. Correspondingly, what the user wants to mark is the first position of the target image displayed by the image 1302 , for example, the position P of the spherical object in FIG. 11B . Furthermore, after the user triggers the operation instruction, the first position information determined by the second terminal device can accurately indicate the position P of the spherical object, and there will be no situation where the user mistakenly believes that the mark is wrong due to screen shake.

It can be seen that with this implementation, not only can the user on the side of the second terminal device accurately identify the position of the mark, but also the first position information can accurately indicate the first position of the target object marked by the user, thereby improving the accuracy of position determination , and further, optimize the effect of "hand in hand" remote assistance.

For other related descriptions of step 502, reference may be made to the description of step 102 above, which will not be described in detail here.

Step 503: The second terminal device determines, according to the first position information, that the first position on the target object corresponds to the second position in the virtual coordinate system, and corresponds the first position on the target object to the second position in the virtual coordinate system Sent to the first terminal device; the second position in the virtual coordinate system corresponds to the actual position of the first position on the target object in the real three-dimensional space.

Step 504: The second terminal device sends the second position to the first terminal device.

For the implementation process of steps 503-504, reference may be made to the implementation process of step 403 in FIG. 7, which will not be described in detail here.

Step 505: The first terminal device renders the aforementioned mark on the second image according to the second position, so that the position corresponding to the mark on the second image is consistent with the position of the first position on the target object in the first image ; the second image is an image including the target object displayed by the first terminal device.

For the description of step 505, reference may be made to the description of step 104 above, which will not be described in detail here.

Step 506: The first terminal device displays the mark rendered on the second image.

For the description of step 506, reference may be made to the description of step 105 above, which will not be described in detail here.

The methods illustrated in the foregoing steps 501 to 506 are only schematic descriptions of the present application, and do not constitute limitations to the embodiments of the present application.

Understandably, after the user on the first terminal device side or the user on the second terminal device side triggers to cancel the screenshot, or after the user on the first terminal device side or the user on the second terminal device side triggers the closing of the remote assistance function, the second terminal The video stream sent by the first terminal device continues to be displayed on the device, and the first image is no longer displayed on the second terminal device.

It can be seen that, in the method described in this embodiment, during the period when the user on the side of the second terminal device triggers the mark, the first image displayed by the second terminal device on the main interface of the second terminal device is unchanged. When the user triggers the first position of the target object, the reference is the target object displayed in the first image. In this way, not only can the user accurately identify the position of the marker, but also the user can accurately observe that the triggered position can be accurately associated with the first position of the target object on the first image. Correspondingly, when the first terminal device renders the mark on the second image including the target object displayed on the first terminal device, the position of the mark displayed on the second image can accurately correspond to the object that the user wants to mark. This can improve the performance of "hands-on" remote assistance and optimize the effect of "hands-on" remote assistance.

Optionally, in some embodiments, after determining the first location information in step 502, the second terminal device may send the first location information to the first terminal device. After that, the first terminal device may perform the operations of steps 103 to 105 . Not detailed here.

Optionally, in some embodiments, the second terminal device acquires the field of view information corresponding to the first image. The content displayed by the second terminal device comes from the first terminal device, the posture of the first terminal device has an influence on the position of the target object displayed by the first image, and the posture of the first terminal device may change frequently, that is, the first terminal device shoots The field of view information corresponding to the image will change. Based on this, in some embodiments, the first terminal device may also send the field of view information corresponding to the first image to the second terminal device. Exemplarily, the first terminal device may also send the field of view information corresponding to the first image to the second terminal device when sending the first image, or send the first image when the second terminal device requests the field of view information corresponding to the first image. The corresponding visual field information is sent to the second terminal device.

When shooting a video, the first terminal device can obtain the feature points of the collected object based on the content of the video. For example, when shooting a cube, the first terminal device can generate the feature points of the cube, and these feature points are used to describe the video. For the cube in the image, these feature points are generally concentrated at the intersection of light and dark in the video image. Understandably, the feature points can indicate the corners of the cube, or indicate the sides or faces of the cube. It is understandable that if there are other When an object is created, the feature points of other objects are also generated. The information of the feature point can be sent to the second terminal device together with the first image, or sent according to the request of the second terminal device.

In some embodiments, the first terminal device may not send the information of the feature points to the second terminal device, but after receiving the first image, the second terminal device may obtain the information of the feature points by calculating according to the first image.

Therefore, in some embodiments, in step 502, after receiving the user's trigger on the first image, the second terminal device determines the location information corresponding to the area triggered by the user. Then, the second terminal device may obtain partial information from the determined position information as the first position information according to the field of view information corresponding to the first image and the feature points of the photographed object. Then, the first location information is sent to the first terminal device, and the first terminal device may render and display the aforementioned mark locally according to the first location information. For example, the first terminal device may perform operations from steps 103 to 105 .

Exemplarily, the first image is an image obtained after an object in a three-dimensional space is converted into a two-dimensional space. Exemplarily, as shown in FIG. 14A , a scene where the first terminal device 1403 shoots a video, the first terminal device 1403 shoots the cube 1402 and the background wall 1401 through the camera, the first terminal device 1403 can take a screenshot to obtain the first image, and the first terminal device 1403 The image and the visual field information corresponding to the first image are sent to the second terminal device. The first terminal 1403 can also calculate the feature points of the first image (1411-1416 in FIG. 14B, corresponding to the points 1411-1415 of the cube 1402 and the point 1416 of the background wall 1401 in FIG. 14A, the feature points here are only for illustration , can have more feature points). Understandably, the feature point may or may not be displayed on the display screen of the first terminal device 1403 . The first terminal device 1403 sends the first image and the field of view information corresponding to the first image to the second terminal device, and may also send information about the feature points 1411-1416 of the photographed object. After receiving the data, the second terminal device displays the first image, as shown in FIG. 14B , the feature points 1411-1416 may be displayed on the second terminal device, or may not be displayed. The user on the second terminal device side marks the object to be marked on the first image, for example, the user marks the upper surface of the cube 1402 with the curve 1420 (ie, the third position information). In order to reduce the data sent to the first terminal, the second terminal device can send the feature point information that has an intersection with the curve 1420 to the first terminal device, so the second terminal device can take the intersection 1411 of the curve 1420 and the feature points 1411-1416, 1412, 1413, 1416. Further, the second terminal device can combine the field of view information corresponding to the first image and the feature point information of the photographed object, and the second terminal device can calculate that the user marked the cube 1402 instead of the background wall 1401. Therefore, the feature is removed from the intersection. Point 1416. The second terminal device sends the simplified intersection 1411, 1412, and 1413 to the first terminal device, so that the first terminal device can calculate that what the user marks is the upper surface of the cube. It can be understood that if the second terminal device does not remove the feature 1416, when the first terminal device uses the intersection sets 1411, 1412, 1413, and 1416 to determine the marked object, it can also use the locally stored field of view information corresponding to the first image and the intersection itself to remove the Feature point 1416. If the feature point 1416 is not removed, when the first terminal device determines the marked object, it may happen that 1411, 1412, 1413 and 1416 in the 3D space are connected and displayed, and the user on the first terminal device side will mistakenly think that the second terminal In addition to marking the cube 1402, the user on the terminal device side also marked the background wall 1401. Understandably, the first terminal device does not send the feature point information to the second terminal device, but the second terminal device calculates and obtains the feature point information according to the first image, and then combines the visual field information corresponding to the first image and the curve marked by the user. to get the first location information.

Based on the display method applied to the first terminal device provided by the above embodiment, an embodiment of the present application further provides a terminal device, and the display device is described below with reference to the accompanying drawings.

Referring to FIG. 8 , this figure is a schematic structural diagram of a terminal device provided by an embodiment of the present application. The terminal device 800 provided in this embodiment of the present application may be the first terminal device mentioned in the above embodiments, and the terminal device 800 may include, for example, a first sending unit 801, a first receiving unit 802, a first determining unit 803, and an adding unit 804 , a rendering unit 805 and a display unit 806 .

a first sending unit 801, configured to send a first image to a second terminal device, where the first image is an image including a target object;

A first receiving unit 802, configured to receive first location information from the second terminal device; the first location information corresponds to an operation performed by a user on the second terminal device for the first location on the target object corresponding to the position in the first image;

The first determining unit 803 is configured to determine, according to the first position information, that the first position on the target object corresponds to the second position in the virtual coordinate system; the second position in the virtual coordinate system corresponds to the the actual position of the first position on the target object in the real three-dimensional space;

An adding unit 804 is configured to add a mark at the second position in the virtual coordinate system;

A rendering unit 805, configured to render the mark on the second image, so that the corresponding position of the mark on the second image is consistent with the position of the first position on the target object in the second image ; the second image is an image including the target object displayed on the terminal device;

A display unit 806, configured to display the mark rendered on the second image.

In a possible implementation manner, the first position information is determined according to a screen position corresponding to an operation triggered by the user on the second terminal device and targeting the first position on the target object; the first position information The first image is displayed on the screen of the two terminal devices.

In a possible implementation manner, the first position information describes the relative position of the first position on the target object in the first image; the first position information is based on the screen position, the The position of the display area of the first image on the screen of the second terminal device is determined.

In a possible implementation manner, the first determining unit 803 is specifically configured to:

determining the image position of the first position on the target object in the image coordinate system corresponding to the first image according to the first position information and the image resolution of the first image;

It is determined according to the image position that the first position on the target object corresponds to the second position in the virtual coordinate system.

In a possible implementation manner, the first determining unit 803 is specifically configured to:

Obtain the field of view information corresponding to the first image, where the field of view information corresponding to the first image is used to describe the target position in the virtual coordinate system of the camera that captured the first image when the first image was captured and the target shooting direction of the camera that shoots the first image in the virtual coordinate system;

According to the field of view information and the image position, the position of the target ray in the virtual coordinate system is calculated; the target ray is the end point of the target position and passes through the second position in the virtual coordinate system ray;

Determining the intersection position of the target ray and the virtual object in the virtual coordinate system as the first position on the target object corresponding to the second position in the virtual coordinate system; the virtual object is in the virtual coordinate system The position in the system corresponds to the actual position of the target object in the real three-dimensional space.

Since the terminal device 800 is a terminal device corresponding to the display method performed by the terminal device provided by the above method embodiments, the specific implementation of each unit of the terminal device 800 is based on the same concept as the above method embodiments. Therefore, For the specific implementation of each unit of the terminal device 800, reference may be made to the description part of the display method performed by the terminal device in the above method embodiments, and details are not repeated here.

Based on the position determination method applied to the second terminal device provided by the above embodiment, the embodiment of the present application further provides a terminal device, which is described below with reference to the accompanying drawings.

Referring to FIG. 9 , this figure is a schematic structural diagram of a terminal device provided by an embodiment of the present application. The terminal device 900 provided in this embodiment of the present application may be the second terminal device mentioned in the above embodiments, and the terminal device 900 includes, for example: a second receiving unit 901, a third receiving unit 902, a second determining unit 903, a third A determination unit 904 and a second transmission unit 905 .

A second receiving unit 901, configured to receive a first image sent by a first terminal device, and display the first image on a screen of the terminal device; the first image is an image including a target object;

A third receiving unit 902, configured to receive an operation instruction triggered by a user on the terminal device for the first position on the target object;

The second determining unit 903 is configured to determine first position information; the first position information corresponds to the operation of the user on the terminal device for the first position on the target object and corresponds to the first image s position;

a third determining unit 904, configured to determine, according to the first position information, that the first position on the target object corresponds to the second position in the virtual coordinate system;

The second sending unit 905 is configured to send the first position on the target object corresponding to the second position in the virtual coordinate system to the first terminal device; the second position in the virtual coordinate system corresponds to the second position in the virtual coordinate system. The actual position of the first position on the target object in the real three-dimensional space.

In an implementation manner, the first position information describes the relative position of the first position on the target object in the first image; the terminal device 900 further includes:

The fourth determining unit is configured to, according to the screen position corresponding to the operation on the first position on the target object triggered by the user on the terminal device, and the position of the first image on the terminal device The position of the display area on the screen determines the first position information.

In an implementation manner, the third determining unit 904 is specifically configured to:

receiving the image resolution of the first image sent by the first terminal device;

determining the image position of the first position on the target object in the image coordinate system corresponding to the first image according to the first position information and the image resolution of the first image;

It is determined according to the image position that the first position on the target object corresponds to the second position in the virtual coordinate system.

In an implementation manner, the terminal device 900 further includes:

The fourth receiving unit is configured to receive the information related to the virtual coordinate system sent by the first terminal device, the information related to the virtual coordinate system, including the virtual object corresponding to the target object in the The position in the virtual coordinate system; the position of the virtual object corresponding to the target object in the virtual coordinate system corresponds to the actual position of the target object in the real three-dimensional space.

In an implementation manner, the third determining unit 904 is specifically configured to:

Receive the field of view information corresponding to the first image sent by the first terminal device, where the field of view information corresponding to the first image is used to describe that when the first image is captured, the camera that captured the first image is in the virtual the target position in the coordinate system and the target shooting direction of the camera that shoots the first image in the virtual coordinate system;

According to the field of view information and the image position, the position of the target ray in the virtual coordinate system is calculated; the target ray is the end point of the target position and passes through the second position in the virtual coordinate system ray;

The intersection position of the target ray and the virtual object in the virtual coordinate system is determined as the first position on the target object corresponding to the second position in the virtual coordinate system.

Since the terminal device 900 is a terminal device corresponding to the position determination method performed by the terminal device provided by the above method embodiments, the specific implementation of each unit of the terminal device 900 is the same concept as the above method embodiments, therefore For the specific implementation of each unit of the terminal device 900, reference may be made to the description part of the position determination method performed by the terminal device in the above method embodiments, which will not be repeated here.

An embodiment of the present application further provides an electronic device, the electronic device includes: a memory and at least one processor;

the memory for storing instructions;

The at least one processor is configured to execute the instruction in the memory, and execute the display method provided by the above embodiment and executed by the first terminal device.

An embodiment of the present application further provides an electronic device, the electronic device includes: a memory and at least one processor;

the memory for storing instructions;

The at least one processor is configured to execute the instructions in the memory, and execute the method for determining the location performed by the second terminal device provided in the above embodiment.

It should be noted that, both the terminal device 800 and the terminal device 900 provided by the embodiment of the present application may have the structure described in FIG. 10 , which is a schematic structural diagram of a terminal device provided by the embodiment of the present application.

Referring to FIG. 10 , the terminal device 1000 includes: a processor 1010 , a communication interface 1020 and a memory 1030 . The number of processors 1010 in the terminal device 1000 may be one or more, and one processor is taken as an example in FIG. 10 . In this embodiment of the present application, the processor 1010 , the communication interface 1020 , and the memory 1030 may be connected through a bus system or other manners, wherein the connection through the bus system 1040 is taken as an example in FIG. 10 .

The processor 1010 may be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP. The processor 1010 may further include a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

As known to those skilled in the art, the number of processors 1010 may also be multiple. For example, in a typical mobile phone architecture, the processors may include an application processor (Application Processor, AP), a baseband processor (Baseband Processor) , BP), may further include a Graphics Processing Unit (GPU), in many recent designs a Neural-network Processing Unit (NPU), and optionally an image signal processing unit ISP (Image Signal Processor, ISP). The above processors may be discrete devices or may be integrated on the same chip. Currently, a typical mobile phone system on chip (System on Chip, SoC) usually includes one or more of the above processors.

The memory 1030 may include volatile memory (English: volatile memory), such as random-access memory (random-access memory, RAM); the memory 1030 may also include non-volatile memory (English: non-volatile memory), such as fast memory Flash memory (English: flash memory), hard disk drive (HDD) or solid-state drive (solid-state drive, SSD); the memory 1030 may also include a combination of the above-mentioned types of memory.

The memory 1030 may store the information related to the virtual coordinate system mentioned in the foregoing embodiments.

Optionally, the memory 1030 stores an operating system and programs, executable modules or data structures, or their subsets, or their extended sets, wherein the programs may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and handling hardware-based tasks. When the terminal device 1000 is the terminal device 800 mentioned in the foregoing embodiment, the processor 1010 can read the program in the memory 1030 to implement the display method provided by the embodiment of the present application; when the terminal device 1000 is the foregoing embodiment When referring to the terminal device 900, the processor 1010 may read the program in the memory 1030 to implement the location determination method provided by the embodiment of the present application.

The bus system 1040 may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus system 1040 can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in FIG. 10, but it does not mean that there is only one bus or one type of bus.

Embodiments of the present application further provide a computer-readable storage medium, including instructions, which, when executed on a computer, cause the computer to execute the display method performed by the first terminal device provided by the above embodiments.

Embodiments of the present application further provide a computer-readable storage medium, including instructions, which, when run on a computer, cause the computer to execute the method for determining a location performed by a second terminal device provided by the above embodiments.

Embodiments of the present application also provide a computer program product including instructions, which, when run on a computer, cause the computer to execute the display method executed by the first terminal device provided in the above embodiments.

Embodiments of the present application also provide a computer program product containing instructions, which, when running on a computer, cause the computer to execute the method for determining a location performed by a second terminal device provided by the above embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a first server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: The technical solutions described in the embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (30)

1. A method of displaying, the method comprising:

the method comprises the steps that a first terminal device sends a first image to a second terminal device, wherein the first image is an image comprising a target object;

the first terminal equipment receives first position information from the second terminal equipment; the first position information corresponds to a position in the first image corresponding to the operation of the user on the second terminal device aiming at the first position on the target object;

the first terminal equipment determines that the first position on the target object corresponds to a second position in a virtual coordinate system according to the first position information, and adds a mark at the second position in the virtual coordinate system; a second position in the virtual coordinate system corresponding to an actual position of the first position on the target object in the real three-dimensional space;

the first terminal device renders the mark on a second image, so that the corresponding position of the mark on the second image is consistent with the position of the first position on the target object in the second image; the second image is an image including the target object displayed on the first terminal device;

the first terminal device displays the marker rendered on the second image.

2. The method according to claim 1, wherein the first position information is determined according to a screen position corresponding to an operation, triggered by a user on the second terminal device, for a first position on the target object; the first image is displayed on a screen of the second terminal device.

3. The method of claim 2, wherein the first location information describes a relative location of a first location on the target object in the first image; and the first position information is determined according to the screen position and the position of the display area of the first image on the screen of the second terminal equipment.

4. The method of claim 3, wherein the first terminal device determines that the first position on the target object corresponds to the second position in the virtual coordinate system according to the first position information, and the method comprises:

the first terminal device determines the image position of the first position on the target object in the image coordinate system corresponding to the first image according to the first position information and the image resolution of the first image;

and the first terminal equipment determines a second position of the first position on the target object in the virtual coordinate system according to the image position.

5. The method of claim 4, wherein the first terminal device determines that the first position on the target object corresponds to the second position in the virtual coordinate system according to the image position, and comprises:

the first terminal device acquires view information corresponding to a first image, wherein the view information corresponding to the first image is used for describing a target position of a camera shooting the first image in the virtual coordinate system and a target shooting direction of the camera shooting the first image in the virtual coordinate system when the first image is shot;

the first terminal device calculates the position of a target ray in the virtual coordinate system according to the visual field information and the image position; the target ray is a ray of which the end point is the target position and passes through the second position in the virtual coordinate system;

the first terminal device determines the intersection point position of the target ray and a virtual object in the virtual coordinate system as a second position of the first position on the target object corresponding to the virtual coordinate system; and the position of the virtual object in the virtual coordinate system corresponds to the actual position of the target object in the real three-dimensional space.

6. The method according to claim 1, wherein the first terminal device displays an image shot by the first terminal device, and before the first terminal device sends the first image to the second terminal device, the method further comprises the step that the first terminal device receives a screen capture request of the second terminal device, and the first terminal device captures the current display content to form the first image.

7. A method of position determination, the method comprising:

the method comprises the steps that a second terminal device receives a first image sent by a first terminal device and displays the first image on a screen of the second terminal device; the first image is an image including a target object;

the second terminal equipment receives an operation instruction which is triggered by a user on the second terminal equipment and aims at a first position on the target object, and determines first position information; the first position information corresponds to a position in the first image corresponding to the operation of the user on the second terminal device aiming at the first position on the target object;

the second terminal equipment determines that the first position on the target object corresponds to the second position in the virtual coordinate system according to the first position information, and sends the second position, corresponding to the first position on the target object, in the virtual coordinate system to the first terminal equipment; the second position in the virtual coordinate system corresponds to an actual position of the first position on the target object in the real three-dimensional space.

8. The method according to claim 7, wherein the second terminal device continuously receives and displays the image sent by the first terminal device during the process that the second terminal device receives the operation instruction, triggered by the user, for the first position on the target object on the second terminal device.

9. The method according to claim 7, wherein before the second terminal device receives the first image transmitted by the first terminal device, the method further comprises the second terminal device transmitting a screen capture request to the first terminal device; the first image is formed by a display content screenshot displayed by the first terminal device to the first terminal device.

10. The method according to claim 9, wherein during the process that the second terminal device receives the operation instruction, triggered by the user, for the first position on the target object, on the second terminal device, the second terminal device continuously receives and does not display the image sent by the first terminal device.

11. The method of claim 7, wherein the first location information describes a relative location of a first location on the target object in the first image; the method further comprises the following steps:

and the second terminal equipment determines the first position information according to the screen position which is triggered by the user on the second terminal equipment and corresponds to the operation aiming at the first position on the target object and the position of the display area of the first image on the screen of the second terminal equipment.

12. The method of claim 11,

the second terminal device determines a second position of the first position on the target object in the virtual coordinate system according to the first position information, and the second position includes:

the second terminal equipment receives the image resolution of the first image sent by the first terminal equipment;

the second terminal device determines the image position of the first position on the target object in the image coordinate system corresponding to the first image according to the first position information and the image resolution of the first image;

and the second terminal equipment determines a second position of the first position on the target object in the virtual coordinate system according to the image position.

13. The method according to any one of claims 7-12, further comprising:

the second terminal device receives information related to the virtual coordinate system, which is sent by the first terminal device, wherein the information related to the virtual coordinate system comprises the position of a virtual object corresponding to the target object in the virtual coordinate system; and the position of the virtual object corresponding to the target object in the virtual coordinate system corresponds to the actual position of the target object in the real three-dimensional space.

14. The method of claim 13, wherein the second terminal device determines that the first position on the target object corresponds to the second position in the virtual coordinate system according to the image position, and comprises:

the second terminal device receives visual field information corresponding to the first image sent by a first terminal device, wherein the visual field information corresponding to the first image is used for describing a target position of a camera shooting the first image in the virtual coordinate system and a target shooting direction of the camera shooting the first image in the virtual coordinate system when the first image is shot;

the second terminal device calculates the position of a target ray in the virtual coordinate system according to the visual field information and the image position; the target ray is a ray of which the end point is the target position and passes through the second position in the virtual coordinate system;

and the second terminal equipment determines the intersection point position of the target ray and the virtual object in the virtual coordinate system as a second position of the first position on the target object corresponding to the virtual coordinate system.

15. A position determination method is characterized in that a first terminal device and a second terminal device are in video call, the first terminal device and the second terminal device both display videos acquired by the first terminal device in real time, and the method comprises the following steps:

the second terminal equipment sends a screen capture request to the first terminal equipment;

the second terminal equipment receives a first image sent by the first terminal equipment and displays the first image on a screen of the second terminal equipment; the first image is an image including a target object; the first image is obtained by the first terminal device by capturing a screen of content displayed on a screen of the first terminal device;

the second terminal equipment receives an operation instruction which is triggered by a user on the second terminal equipment and aims at a first position on the target object, and determines first position information;

and the second terminal equipment sends the first position information to the first terminal equipment.

16. The method of claim 15, further comprising:

the second terminal equipment receives the visual field information corresponding to the first image;

the second terminal device receives an operation instruction which is triggered by a user on the second terminal device and aims at a first position on the target object, and the determining of the first position information comprises the following steps:

the second terminal equipment determines third position information according to the operation instruction,

and the second terminal determines the first position information according to the third position information and the view field information corresponding to the first image.

17. A terminal device, characterized in that the terminal device comprises:

the first sending unit is used for sending a first image to a second terminal device, wherein the first image is an image comprising a target object;

a first receiving unit, configured to receive first location information from the second terminal device; the first position information corresponds to a position in the first image corresponding to the operation of the user on the second terminal device aiming at the first position on the target object;

the first determining unit is used for determining a second position, corresponding to the first position on the target object, in the virtual coordinate system according to the first position information; a second position in the virtual coordinate system corresponding to an actual position of the first position on the target object in the real three-dimensional space;

an adding unit for adding a mark at a second position in the virtual coordinate system;

a rendering unit for rendering the marker on a second image such that a corresponding position of the marker on the second image coincides with a position of the first position on the target object in the second image; the second image is an image including the target object displayed on the terminal device;

a display unit for displaying the marker rendered on the second image.

18. The terminal device according to claim 17, wherein the first location information is determined according to a screen location corresponding to an operation, triggered by the user, on the second terminal device for the first location on the target object; the first image is displayed on a screen of the second terminal device.

19. The terminal device of claim 18, wherein the first location information describes a relative location of a first location on the target object in the first image; and the first position information is determined according to the screen position and the position of the display area of the first image on the screen of the second terminal equipment.

20. The terminal device of claim 19, wherein the first determining unit is specifically configured to:

determining the image position of the first position on the target object in the image coordinate system corresponding to the first image according to the first position information and the image resolution of the first image;

and determining a second position of the first position on the target object corresponding to the virtual coordinate system according to the image position.

21. The terminal device of claim 20, wherein the first determining unit is specifically configured to:

acquiring view information corresponding to a first image, wherein the view information corresponding to the first image is used for describing a target position of a camera shooting the first image in the virtual coordinate system and a target shooting direction of the camera shooting the first image in the virtual coordinate system when the first image is shot;

calculating the position of a target ray in the virtual coordinate system according to the visual field information and the image position; the target ray is a ray of which the end point is the target position and passes through the second position in the virtual coordinate system;

determining the intersection point position of the target ray and a virtual object in the virtual coordinate system as a second position of the first position on the target object corresponding to the virtual coordinate system; and the position of the virtual object in the virtual coordinate system corresponds to the actual position of the target object in the real three-dimensional space.

22. A terminal device, characterized in that the terminal device comprises:

the second receiving unit is used for receiving a first image sent by first terminal equipment and displaying the first image on a screen of the terminal equipment; the first image is an image including a target object;

a third receiving unit, configured to receive an operation instruction, triggered by the user on the terminal device, for a first location on the target object;

a second determination unit for determining the first position information; the first position information corresponds to a position in the first image corresponding to the operation of the user on the terminal equipment aiming at the first position on the target object;

a third determining unit, configured to determine, according to the first position information, a second position, in the virtual coordinate system, where the first position on the target object corresponds to;

the second sending unit is used for sending the first position on the target object to a second position corresponding to the virtual coordinate system in the first terminal device; the second position in the virtual coordinate system corresponds to an actual position of the first position on the target object in the real three-dimensional space.

23. The terminal device of claim 22, wherein the first location information describes a relative location of a first location on the target object in the first image; the device further comprises:

a fourth determining unit, configured to determine the first location information according to a screen location corresponding to an operation, triggered by the user on the terminal device, for a first location on the target object and a location of a display area of the first image on a screen of the terminal device.

24. The terminal device of claim 22,

the third determining unit is specifically configured to:

receiving the image resolution of the first image sent by the first terminal equipment;

determining the image position of the first position on the target object in the image coordinate system corresponding to the first image according to the first position information and the image resolution of the first image;

and determining a second position of the first position on the target object corresponding to the virtual coordinate system according to the image position.

25. The terminal device according to any of claims 22-24, wherein the apparatus further comprises:

a fourth receiving unit, configured to receive information related to the virtual coordinate system, which is sent by the first terminal device, where the information related to the virtual coordinate system includes a position of a virtual object corresponding to the target object in the virtual coordinate system; and the position of the virtual object corresponding to the target object in the virtual coordinate system corresponds to the actual position of the target object in the real three-dimensional space.

26. The terminal device of claim 25, wherein the third determining unit is specifically configured to:

receiving visual field information corresponding to the first image sent by a first terminal device, wherein the visual field information corresponding to the first image is used for describing a target position of a camera shooting the first image in the virtual coordinate system and a target shooting direction of the camera shooting the first image in the virtual coordinate system when the first image is shot;

calculating the position of a target ray in the virtual coordinate system according to the visual field information and the image position; the target ray is a ray of which the end point is the target position and passes through the second position in the virtual coordinate system;

and determining the intersection point position of the target ray and the virtual object in the virtual coordinate system as a second position of the first position on the target object corresponding to the virtual coordinate system.

27. An electronic device, characterized in that the electronic device comprises: a memory and at least one processor;

the memory to store instructions;

the at least one processor, configured to execute the instructions in the memory, to perform the method of any of claims 1-6.

28. An electronic device, characterized in that the electronic device comprises: a memory and at least one processor;

the memory to store instructions;

the at least one processor, configured to execute the instructions in the memory, to perform the method of any of claims 7-16.

29. A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any of claims 1-6 above.

30. A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any of claims 7-16 above.

Images