CN105046752A - Method for representing virtual information in a view of a real environment - Google Patents

Abstract

A method for depicting image information in a view of a real environment, comprising: capturing a first plurality of images of a real environment by a first mobile device; storing the first plurality of images in a database; capturing the first plurality of images by a second mobile device a second image of the real environment; matching the first plurality of images relative to the first plurality of images, wherein the matching includes performing a bundle adjustment; for each corresponding image of the first plurality of images, determining to allow drawing conclusions about the first pose data about the position and orientation relative to the frame of reference in which the corresponding images were taken; comparing the second image with the images in the first plurality of images; for the delineation in the first plurality of images image, according to the matching result and the first pose data associated with the depicted image, determine an overlay position in the second image; and display the depicted image overlaid on the determined overlay position in the second image.

Description

Method for rendering virtual information in view of real environment

This application is a divisional application of the Chinese invention patent application with the application number 201080045773.1, the application date is October 11, 2010, and the invention title is "Method for Depicting Virtual Information in the View of Real Environment".

technical field

The present invention relates to a method for rendering virtual information in a view of a real environment.

Background technique

Augmented reality (AR) is a technology that overlays virtual data on reality, so this technology helps to realize the association between data and reality. The use of mobile AR systems is known in the art. Over the past few years, high-performance mobile devices (eg, smartphones) have become suitable for AR applications. These devices also have relatively large color displays, built-in cameras, superior processors, and additional sensors (eg, orientation sensors and GPS). In addition, the location of the device can be approximated through the wireless network.

In the past, there were various projects realized on mobile devices using AR. First, special optical targets for determining the position and orientation of the device are used. Regarding AR which is also available for larger areas (hence it is also called larger area AR), hints for realistically depicting objects in conjunction with HMDs (Helm Mounted Displays) [3] have been published. In recent times there are also methods using GPS and orientation sensor systems of modern devices [1, 2, 4, 5].

However, the methods published so far have the disadvantage that they do not allow easy fusion of other users in the AR scene. In addition to this, most systems based on GPS and compass have the disadvantage that these devices have to be convincingly presented, but there can be large inaccuracies.

US 2009/0179895A1 describes a method for mixing three-dimensional annotations or annotations in images of the real environment ("street view"). By means of a selection box in the image, the user selects where to blend annotations. Thereafter, the selection box is projected onto the three-dimensional model in order to determine the position of the annotation with respect to the image. Additionally, location data corresponding to projections on the three-dimensional model are determined and associated with user-entered annotations. This annotation is stored in the server's database along with the location data, from which it can be blended with another image of the real environment.

The term "tagging" is used generally and hereinafter to describe the user's enrichment of reality with additional information. Methods implemented so far in conjunction with tagging include placing objects in a map view (e.g., Google Maps), taking pictures of location points, storing these images with additional comments, and generating text at specific location points information. Disadvantageously, distant viewers and users can no longer gain AR access to interactive scenes in the world. Only so-called screenshots (screen images) of that AR scene are seen, but no longer changed.

The object of the present invention is to indicate a method of depicting virtual information in a view of the real environment, which allows users to interactively view other user-generated AR image scenes through augmented reality, and guarantees high accuracy and user friendliness when implemented .

Contents of the invention

According to a first aspect of the present invention, there is provided a method for rendering virtual information in a view of a real environment, the method comprising the steps of: providing, in a database of a server, a global position and orientation with a geographic global coordinate system at least one virtual object, and first pose data allowing conclusions to be drawn about the global position and orientation of said virtual object; capturing at least one image of a real environment via a mobile device, and providing second pose data, wherein said second Pose data allows conclusions to be drawn as to where and in what direction the image was taken in the geographic global coordinate system; display the image on the display of the mobile device; access all data in the server's database the virtual object, and based on the first posture data and the second posture data, locate the virtual object in the image displayed on the display; by performing corresponding positioning in the image displayed on the display, operate the virtual object or add another virtual object; in the database of the server, provide the operated virtual object according to the location in the image and the modified first posture data, or other information according to the location in the image Both the virtual object and the third pose data, the modified first pose data and said third pose data allow conclusions to be drawn about the global position and orientation of the manipulated virtual object or of the further manipulated object. In this aspect, the image and the second pose data may be provided eg on the server.

According to a further aspect of the present invention, there is provided a method for rendering virtual information in a view of a real environment, the method comprising the steps of: providing, in a database of a server, at least a virtual object, and first pose data allowing conclusions to be drawn about the global position and orientation of said virtual object; at least a view and second pose data, wherein the second pose data allows conclusions to be drawn about where and what orientation the data glasses are located in the geographic global coordinate system; accessing the virtual object, and based on the first pose data and the second pose data, position the virtual object in the view; through the corresponding positioning in the view, operate the virtual object or add another virtual object; In the database of the server, the manipulated virtual object and the modified first pose data according to the positioning in the view, or the additional virtual object and the third pose data according to the positioning in the view are provided, so that Both the modified first pose data and said third pose data allow conclusions to be drawn about the global position and orientation of the manipulated or further virtual object.

In an embodiment of the present invention, said mobile device or said data glasses comprise (or are connected to) a unit for generating said second pose data.

For example, the gesture data may each include three-dimensional values for position and orientation. Furthermore, the orientation of the image of the real environment can be specified independently of the surface of the earth.

According to another embodiment of the present invention, which image among the several images of the real environment or which view among the several views of the real environment is stored on the storage location of the server, which virtual object among the several virtual objects has been provided There is pose data.

When determining the position of a mobile device eg by means of a GPS sensor (GPS: Global Positioning System), it may happen that the position of the mobile device is determined only in a relatively inaccurate manner due to inaccuracies of the sensor or inherent inaccuracies of the GPS. This can be the result of placing the blended virtual object in an image that is inaccurately relative to the geographic global coordinate system, so that in other images or views that have different viewing angles, the virtual object blended here has a corresponding relation to reality. The displacement mode is shown.

In order to achieve a more accurate depiction of a virtual object or its position in an image of the real environment, an embodiment of the method according to the invention comprises the steps of: providing a reference database with a reference view of the real environment and pose data allowing drawing a conclusion as to where and in what direction the camera took each reference view in a geographic global coordinate system; comparing at least one real object shown in said image with at least a portion of a real object contained in at least one of said reference views Comparing, the second pose data of the image matches the pose data of the at least one reference view; as a result of the matching, modifying the second pose data based on at least a part of the pose data of the at least one reference view at least part of .

Furthermore, another embodiment includes: modifying at least a part of the first pose data of the virtual object located in the image as a result of matching the second pose data of the image with the pose data of the at least one reference view .

Further developments and embodiments of the invention can be obtained through the appended claims.

Aspects and embodiments of the invention are explained in more detail below by way of the figures shown in the drawings.

Description of drawings

Figure 1A shows a plan view of a schematic arrangement of a first exemplary embodiment of a system setup that can be used to carry out the method according to the invention.

Figure 1B shows a plan view of a schematic arrangement of a second exemplary embodiment of a system setup that can be used to carry out the method according to the invention.

Fig. 1C shows a schematic view of a possible data structure of an embodiment of a system for performing the method according to the invention.

Fig. 2 shows a schematic view of an overview of participating coordinate systems, according to one embodiment of the invention.

Fig. 3 shows an exemplary flow of a method according to an embodiment of the present invention.

Fig. 4 shows an exemplary flow of a method according to another embodiment of the present invention, in particular supplemented with an optional measure for improving image pose.

Fig. 5 shows an example scene of a real environment with virtual objects placed therein without implementing pose refinement.

Fig. 6 shows an example scene of a real environment with virtual objects placed therein after implementing pose refinement.

FIG. 7A shows an exemplary map view of the real world in which virtual objects are placed.

Figure 7B shows an exemplary perspective view of the same scene as Figure 7A.

detailed description

FIG. 1A shows a plan view depicting a schematic arrangement of a first exemplary embodiment of a system setup that can be used to carry out the method according to the invention.

In the view of FIG. 1A , a user wears a head-mounted display system (“Helm Mounted Display”, abbreviated as HMD) as a display device, wherein the system includes a display 21 as part of a system setup 20 . At least a portion of system arrangement 20 may be considered as a mobile device comprising one or more interconnected components, as described in more detail below. These components may be connected to each other by wired connections and/or wirelessly. Furthermore, some of these components (eg computer 23 ) may also be provided as stationary components, ie they do not move with the user. For example, the display 21 may be in the form of so-called optical see-through data glasses ("optical see-through display" in which reality can be viewed through data glasses of a semi-transparent structure) or in the form of so-called video see-through data glasses ( "Visual see-through display" in which reality is depicted on a screen worn in front of the user's head) in the form of data glasses in which virtual information provided by the computer 23 can be mixed in a known manner. The user then views objects of the real environment 40 augmented with the blended virtual information 10 in a view 70 of the real world (which may be viewed through or on the display 21 ) in an viewing angle or aperture angle 26 ( For example, so-called point-of-interest objects related to the real world, which are simply referred to as POI objects). The virtual object 20 is blended such that the user perceives the object in a manner as if the object is arranged at an approximate position in the real environment 40 . Additionally, this position of virtual object 10 may also be stored as a global position in a geographic global coordinate system (eg, that of the earth), as described in more detail below. In this way, the system arrangement 20 constitutes a first embodiment of a generally known augmented reality system, which system can be used for the method according to the invention.

The display 21 may have further sensors 24 (eg rotation sensors, GPS sensors or ultrasonic sensors) and a camera 22 mounted thereon for optical tracking and taking one or more images (so-called "views"). The display 21 may be translucent, or have a realistic image provided by the camera image of the camera 22 . With a translucent display 21, a calibration between the user's eyes 25 and the display 21 must be performed. This process, which is called see-through calibration, is well known in the art. This calibration can advantageously determine the pose of the eye relative to the camera 22 at the same time. A video camera may be used to capture or record the views for access by other users, as described in more detail below. Pose is usually understood as the position and orientation of an object relative to a reference coordinate system. For determining the pose, there are various methods described in the prior art, which are known to the skilled person. Advantageously, there may also be a position sensor installed on the display 21 or anywhere on the user's body or in the computer 23, e.g. GPS sensor (GPS: Global Positioning System) based on longitude, latitude and altitude). Pose determination of any part of the system setup can be achieved using the principles provided to draw conclusions about the user's position and viewing direction.

The view of Fig. IB shows another exemplary system setup 30 as typically found in, for example, modern mobile telephones (so-called "smartphones"). A display device 31 (for example in the form of a screen or display), a computer 33, a sensor 34 and a camera 32, constitute a system unit housed in a common housing, eg a mobile phone. At least a portion of the system arrangement 30 may be considered as a mobile device including one or more of the mentioned components. These components may be housed in a common housing, or may be distributed (partially) and connected to each other by wired connections and/or wirelessly.

The view of the real environment 40 is provided by a display 31 displaying an image 50 of the real environment 40 captured by the camera 32 at an observation angle and using an aperture angle 36 . For augmented reality applications, a photographic image 50 may be displayed on a display 31 and augmented with additional virtual information 10 (e.g., POI objects related to the real world) having a specific location relative to reality, similar to that described in FIG. 1A . In this way, the system arrangement 30 constitutes another embodiment of what is commonly known as an Augmented Reality (AR) system.

A calibration similar to that described with reference to FIG. 1A is used to determine the pose of the virtual object 10 with respect to the camera 32 in order to make this same content accessible to other users, as described in more detail below. For pose determination there are various methods described in the prior art which are known to the skilled person. Advantageously, on the mobile device (especially when the system arrangement 30 has the form of a unit) or anywhere on the user's body or in the computer 33, there may also be a connected position sensor, for example a GPS sensor 34, so that Allows for geographic location determination (eg, according to longitude and latitude) of system settings 30 in the real world 40 . In some cases, no camera is needed for pose determination, for example when pose is determined by GPS and orientation sensors alone. Basically, pose determination is appropriate for any part of the system setup as long as conclusions can be drawn about the user's position and viewing direction.

Basically, the present invention can be conveniently used in all forms of AR. For example, it does not matter whether the representation is implemented using a so called optical see-through mode with a translucent HMD or a video see-through mode with a camera and a display screen.

Basically, the invention can also be used in conjunction with stereoscopic displays in which the video see-through method advantageously uses two cameras, one for recording one image stream for each eye. In any case, virtual information items can be calculated for each eye individually and stored in pairs on the server.

Basically, the processing of the steps of different parts described below can be distributed to different computers via a network. Therefore, a client/server architecture or a more client-based solution is possible. In addition, the client or the server may also include some computing units, for example, some CPUs or dedicated hardware components, such as commonly known FPGAs, ASICs, GPUs or DSPs.

To allow AR, the pose (position and orientation) of the camera in space is required. This can be accomplished in a number of different ways. For example, attitude in the real world can be determined by using only GPS and an orientation sensor with an electronic compass (as installed eg in some modern mobile phones). However, the uncertainty of this pose is very high. Therefore, other methods can also be used, such as optical initialization and tracking or a combination of optical methods with GPS and orientation sensors. WLAN positioning can also be used, or RFID (tags or chips for "Radio Frequency Identification") or optical tags can support the positioning process. As mentioned above, a client/server based approach is also possible here. Specifically, the client can request location-specific information needed for light tracking from the server. For example, this information may be a reference image of the surrounding environment with pose information and depth information. An optional embodiment of the invention in this aspect proposes that the pose of the placed virtual object in the world, in particular the pose of the view on the server, can be enhanced based on this information.

Besides, the present invention can also be installed (or carried) in a vehicle, aircraft or ship using a monitor, HMD or head-up display.

Basically, virtual objects such as points of interest ("POIs") can be created for a wide variety of different forms of information. Some examples are given below: An image that can represent a location using GPS information. Information can be automatically extracted from the Internet. For example, this could be a company or restaurant website with an address or a page giving a rating. Users can deposit text, images or 3D objects at specific locations and make them available to others. Information pages such as wikipedia can be searched for geographic information, and these pages can be accessed as POIs. POIs can be automatically generated according to the search and browsing behavior of the user of the mobile device. Other locations of interest may be displayed, such as underground transit or bus stops, hospitals, police stations, doctors, real estate advertisements, or health clubs.

The user can store these information items in the image 50 or view 70 (compare FIGS. 1A and 1B ) as virtual objects 10 at specific locations in the real world 40, and make other objects with locations corresponding to the respective locations User accessible. Then, in the accessible view or image of the real world, other users can for example manipulate this information (where the information is blended according to its position), or also add further virtual objects. This will be described in more detail below.

Fig. 1C first shows the data structure used according to the embodiment of the present invention, which will be briefly explained below.

A view is a captured view of the real world, in particular a view (compare view 70 according to FIG. 1A ), an image (compare image 50 according to FIG. 1B ) or a sequence of images (movie or motion picture). Associated with the views (image 50/view 70) are camera parameters describing the light properties of the cameras 22, 32 (eg, with respect to aperture angle, focus shift or image distortion), and are associated with image 50 or view 70, respectively. Among other things, a view has pose data associated with it, which describes the position and orientation of the image 50 or view 70 with respect to the earth. To this end, a geographic global coordinate system is associated with the earth, so that a real-world geographic global position determination can be provided, eg in terms of longitude and latitude.

The placed model is a graphically displayable virtual object (compare object 10 according to FIGS. 1A, 1B ), wherein this virtual object also has pose data. A placed model may represent, for example, an instance of a model in a model database, ie reference the model. Advantageously, each virtual model 10 is placed in the world 40 according to the stored view 50 or 70 (if so). This can be used to improve pose data, as described in more detail below. A scene is formed with a combination of views 50, 70 and 0 to n placed models 10, optionally containing a generation date. Furthermore, all or a portion of the data structure may be linked with metadata. For example, creator, date, frequency of images/views, rating and keywords can be stored.

In the following aspect of the invention, more detail will be described with respect to the embodiment according to FIG. 1B , wherein an image 50 is captured by a camera 32 and blended with a virtual object 10 on a display 31 for viewing by a viewer. However, the skilled person can easily convert the statement of this aspect to the embodiment using the HMD according to Fig. 1A.

Figure 2 gives an overview of the participating coordinate systems, according to an embodiment of the present invention. In one aspect, an earth coordinate system 200 (which in this embodiment is represented by a geographic global coordinate system) is used that constitutes the link elements. The reference numeral 201 is used in FIG. 2 to indicate the surface of the earth. To specify a geographic global coordinate system (e.g., Earth Coordinate System 200), various standards known to those of ordinary skill in the art are specified (e.g., WGS84; NMA-National Mapping Agency: Department of Defense World Geodetic System 1984; January 2000 Third Edition Technical Report TR8350.2). Furthermore, the camera coordinate system provides a link between the displayed virtual object 10 and the image 50 . The pose of camera 32 and image 50 in earth coordinate system 200 , pose P50_10 of object 10 relative to image 50 (“pose model in image”) can be calculated by transformations known to the person skilled in the art. For example, a global image pose PW50 ("pose image in the world") is computed by GPS and/or orientation sensors. From the poses PW50 and P50_10, a global pose PW10 (“pose model in the world”) of the virtual object 10 can then be calculated.

In a similar manner, the pose P60_10 of the object 10 relative to the image 60 can be calculated from the pose of the second image 60 with another global pose PW60 in the earth coordinate system 200 ("pose model in image 2"). For example, via GPS and/or orientation sensors, a global image pose PW60 ("Pose image 2 in the world") is also calculated.

In this way, it is possible to place a virtual object 10 in a first image (image 50) and view the virtual object 10 in a second image (image 60) located at a nearby location on Earth, but from a different viewing angle. watch. For example, a first user places object 10 in first image 50 with pose PW10. When the second user uses his mobile device, a view is generated from the image 60, if the image 60 covers in the aperture angle or viewing angle a part of the real world including the global position of the pose PW10, the virtual object 10 placed by the first user automatically blended into image 60 at the same global position corresponding to pose PW10.

In the following, aspects and embodiments of the present invention are described in more detail by means of the flowcharts of FIGS. 3 and 4 in conjunction with other figures.

Fig. 3 shows an exemplary flow of a method according to an embodiment of the present invention. In a first step 1.0, data about the world is generated. These data can be extracted eg from the Internet, or can be generated by the first user using a camera (FIG. 1B) or an HMD with a camera (FIG. 1A). To do this, the user takes a view (image or captured view) in step 1.0 by determining the position and orientation (pose) in the world (step 2.0). For example, this can be achieved using GPS and a compass. Optionally, information about the uncertainty in the generation of the data may additionally be recorded.

When this view (image or captured view) exists, the user can advantageously place virtual objects in this view directly on his mobile device (step 3.0). Advantageously, the object is placed and manipulated in the camera coordinate system. In this case, in step 4.0, the global pose of the virtual object (or object) in the world (e.g., relative to the coordinate system 200) is calculated from the global pose of the view and the pose of the object in the camera coordinate system . This can happen on client 1 as well as server 2.

A client is a program on a device that contacts another program on a server to use its services. The underlying client-server model allows tasks to be distributed to different servers in a computer network. Instead of solving one or more of its specific tasks, the client has them solved by the server, or receives corresponding data from the server that provides services for this effect. Basically, most of the steps of the system can be performed on the server as well as on the client. Using clients with a high computing capacity, it is for example advantageous to have them perform as many calculations as possible and thus free up servers.

In step 5.0, these items of information from step 4.0 are then stored in the database 3 of the server 2, as fully described with reference to FIG. 1C. In step 6.0, the same user or another user on another client takes an image of the real environment (or views a specific part of the environment by means of an HMD), and then the location of the viewed real environment is loaded from the server 2 in step 6.0. Data stored in 5.0. Loading and displaying location-related information using augmented reality and databases advantageously equipped with geospatial functionalities is known in the art. The user now views previously stored information from a previous store or a new viewing angle and is able to make changes (manipulate existing virtual information and/or add new virtual information), which are then stored on the server 2 . Here, the user does not necessarily have to be online when the user sits in his office, for example at an internet-capable client, but can use a previously stored view as a window into reality.

In the example of FIGS. 1 and 2 , the user thus provides or generates a virtual object 10 on a database 3 of a server 2 which has a global position and orientation in a geographic global coordinate system 200 and which allows for viewing of the virtual object 10 The global position and orientation are concluded from the attitude data (attitude PW10). The user or another user captures at least one image 50 of the real environment 40 with the mobile device 30 and pose data (pose PW50) that allows to determine where and in what direction the image 50 was taken in the geographic global coordinate system 200 draw conclusions. The image 50 is displayed on the display 31 of the mobile device. The virtual object 10 is accessed in the database 3 of the server, and then based on the pose data of the poses PW10 and PW50, the virtual object 10 is positioned in the image 50 displayed on the display. Subsequently, the virtual object 10 can be manipulated (e.g., displaced) by corresponding positioning in the image 50 displayed on the display (cf. arrow MP of FIG. A dummy object11.

Subsequently, the manipulated virtual object 10 and the modified pose data (modified pose PW10) according to the positioning in the image 50 or this further virtual object 11 and the positioning according to the image 50 are stored in the database 3 of the server 2 Its pose data, the modified pose data PW10 and the pose data of the new virtual object 11 both allow conclusions to be drawn about the global position and orientation of the manipulated object 10 or the further virtual object 11 with respect to the coordinate system 200 .

In some cases, it happens that the server cannot be reached, so new scenes cannot be stored. Advantageously in this case, the system can function to provide caching of this information until the server is available again. In one embodiment, if the network connection to the server fails, the data to be stored on the server is cached on the mobile device and sent to the server when the network connection becomes available again.

In another embodiment, a user can obtain an atlas of scenes in a real-world area (e.g., his surrounding area or a local area), which allows the user to view them in a list sorted by proximity, or on a map, or using enhanced Choose realistically.

In another embodiment, the image or virtual information has a unique identification characteristic (for example, a unique name), and the existing image or virtual information (which may be virtual model data or view) on the client or the mobile device does not need to be reused. Instead of downloading from the server, it loads from the local data store.

Fig. 4 shows an exemplary flow of a method according to another embodiment of the present invention, in particular supplemented with an optional measure for improving image pose. The method includes steps 1.0 to 6.0 in FIG. 3 . Furthermore, in steps 7.0 to 8.0 of FIG. 4 , for example by means of light methods, the pose of the view (image or captured view) is subsequently raised, since information is advantageously stored via which view the virtual information is placed, so the The posture of information is also correct. Alternatively, the view can be improved immediately after the view has been generated on the client 1 by providing light-tracing reference information for that view, or by providing the client 1 with a view with a similar pose from the reference database 4 of the server 2. attitude. Alternatively, it is also possible to achieve the accuracy of the view and directly store it in the correct way before computing the pose of the placed virtual object (step 4.0). However, the advantage of the latter approach is that existing reference data does not have to be available for all locations, so correctness can also be performed for these views as long as reference data is available.

Of course, other views used as reference data may also be used, especially when multiple views are available for a location. This method, called bundle adjustment, is known in the art, for example, in ACM Transactions on Mathematical Software, Vol. 36, No. 1 Article 2, published March 2009, MANOLOSI.A., LOURAKIS and ANTONISA.ARGYROSA:SBA:ASoftwarePackageforGenericSparseBundleAdjustment This is described. In this case, the similar 3D position of the point, the pose of the view, and the intrinsic camera parameters can be optimized. Therefore, the method according to the invention also provides the ability to generate one's own world model in order to use this data in general. For example, for masking models that support depth sensing or for real-time light tracing.

Fig. 5 shows an exemplary scene of a real environment with virtual objects placed therein without pose refinement taking place so far. Figure 5 shows a possible situation before correction. In the image displayed by the display 31 of the device 30 , a virtual object 10 (eg a review of a restaurant) is placed relative to real objects 41 , 42 (representing eg a restaurant building), as seen from the mobile device 30 . Based on incorrect or inaccurate GPS data, images and objects 10 are stored with incorrect world coordinates in a manner corresponding to incorrect or inaccurately determined camera pose data P30-2. This results in object 10-2 being stored in a correspondingly incorrect manner. This is no problem in the captured image itself. However, errors can become apparent when viewing the virtual object 10 eg on a map or on another image.

If an image is generated with true or correct camera pose data P30-1, the virtual object 10 will be displayed at a position in the image, as indicated by the depiction of the virtual object 10-1, and also with the user generated virtual object 10-1. The way the object is viewed. However, the incorrectly stored virtual object 10-2 is shown in another image as being displaced from the real position of the virtual object 10 with the wrong camera pose P30-2 being displaced from the real camera pose P30-1 degree. Thus, the incorrectly stored depiction of the virtual object 10-2 in the image of the mobile device 30 does not correspond to the generated true location of the user in the previous image.

In order to improve the accuracy of the depiction of virtual objects and their positions in images of the real environment, an embodiment of the method according to the invention comprises the following steps: providing a reference database 4 with reference views of the real environment and pose data, wherein the pose The data allows conclusions to be drawn as to where in the geographic global coordinate system 200 and in what orientation the camera took each reference view. Subsequently, comparing at least a portion of the real object shown in the image with at least a portion of the real object contained in at least one of the reference views enables matching of pose data of the image with pose data of the at least one reference view. Thereafter, as a result of the matching, at least a part of the pose data of the image is modified based on at least a part of the pose data of the respective reference view.

Furthermore, in a further embodiment at least a part of the pose data of the virtual object located in the image is modified as a result of the matching of the pose data of the image with the pose data of the respective reference view.

FIG. 6 shows an exemplary scene similar to the real environment of FIG. 5 with a virtual object 10-1 placed therein after pose improvement has occurred. On the one hand, Fig. 6 shows the mechanism for recognizing image features in an image, and on the other hand, shows the corresponding correctness of image pose and object pose. Specifically, image features 43 (eg, different features of real objects 41 and 42 ) are compared with corresponding features of reference images of reference database 4 and matched (referred to as "matching" of image features).

The virtual object 10 is now also correctly depicted in the other image (which has the correct pose), or can be placed correctly. The well-placed expression states that users do misjudge the height of objects placed on the ground when placing virtual objects in perspective. By having two images overlapping in part of the recorded reality, it is possible to extract the ground plane, repositioning objects placed in such a way that they are all on the ground, but in this image where they were originally placed, as if still almost at the same location.

FIG. 7A shows an exemplary map view of the real world in which virtual objects are placed, while FIG. 7B shows an exemplary perspective view of the same scene as FIG. 7A. 7A and 7B are used to specifically depict user-assisted determination of camera pose. It is useful, for example, when using a mobile device that is not equipped with a compass to obtain a rough estimate of the viewing direction. To this end, the user can take an image 50 and place the virtual object 10 with respect to the real object 41 in the usual way, as shown in FIG. 7B . Thereafter, the user may be prompted to display the location of the placed object 10 again on the map 80 or virtual view 80 of the world, as shown in FIG. 7A . Based on the connection between the GPS location of image 50 and the object location of object 10 on map 80, the orientation (orientation) of image 50 in the world can then be calculated or corrected. When the mobile device also does not have GPS, the process can also be performed using two virtual objects or one virtual object and an indication of the current location. In addition, the "field of view" of the last image can also be indicated to the user (indicator 81 of the comparison image part), as exemplified in Fig. 7A, the user can interactively move the "field of view" in the map for correctness, and to redirect. Here, the aperture angle of the "field of view" is displayed according to intrinsic camera parameters.

Specifically, according to this embodiment, the method comprises the steps of: providing a map view (cf. map view 80 ) on the display of the mobile device, providing the user with the option to select the viewing direction when the image is taken. In this way, it is possible to select in the map which viewing direction the user is facing with the camera at a particular moment.

According to another embodiment of the present invention, the method includes the additional step of placing a virtual object in the image of the real environment and in the map view provided by the display of the mobile device, according to the determined position of the image and the position of the virtual object in the Provides a location in the map view to orient the image. Thus, virtual objects can be placed on a map as well as in a see-through image of the real environment, which allows conclusions to be drawn about the direction of the user.

In order to also allow other users to view and edit images of real environments enhanced with virtual objects from a distance (for example, on a client communicating with a server such as the Internet), an embodiment of the present invention provides further steps Methods:

At least one image of the real environment and its pose data are provided on the server's database. Thereafter, the image of the real environment is accessed on the server, and the image is sent to the client device, so that the image is displayed on the client device. The user operates the virtual object or adds another virtual object by corresponding positioning in the image of the real environment displayed on the client device. On the server's database, provide the manipulated virtual object and its modified pose data according to the location in the image displayed by the client device, or another virtual object according to the location in the image displayed by the client device and its ( new) pose data, wherein either the modified pose data or the new pose data allow conclusions to be drawn about the global position and orientation of the manipulated virtual object or the further virtual object in the image displayed by the client device. Thus, using "remote access" of the client device, the AR scene in the image can be modified or enhanced with additional virtual information and written back to the server. Due to the newly stored virtual information of the operation or the global position of the new virtual information, other users can retrieve the position by accessing the server, and watch it in the AR scene corresponding to the global position.

Based on the above, the method in another embodiment includes the following additional steps: accessing the image of the real environment on the server, and sending it to the second client device so that the image can be viewed on the second client device, storing The virtual objects provided on the server are fetched, wherein the view of the image on the second client device displays those virtual objects whose global positions are within the real environment displayed in the view of the image on the second client device. In this way, a viewer can observe on another client device a scene displaying those virtual objects that other users have previously placed in corresponding locations (i.e., their global positions are at the location on that client device). within the real environment shown in the view of the image). In other words, the viewer sees from his point of view those virtual objects that other users have previously placed in the viewable range of the view.

[1]ARWikitude.http://www.mobilizy.com/wikiitude.php.

[2]Enkin.http://www.enkin.net.

[3] S. Feiner, B. MacIntyre, T. Höllerer, and A. Webster. Atouring machine:

Prototyping 3d mobile augmented reality systems for exploring the urban environment. In Proceeding of the 1st International Symposium on Wearable Computers, pages 74–81, 1997.

[4]SekaiCamera.http://www.tonchidot.com/product-info.html.

[5]layar.com.

Claims (22)

1., for a method for rendering image information in the view of true environment, comprise the following steps:

1a) by more than first image of at least one the first mobile device shooting true environment;

1b) described more than first image is stored at least one database;

The second image of described true environment 1c) is taken by the second mobile device;

1d) access described more than first image stored at least one database described;

1e) described more than first image is mated relative to described more than first image; Wherein, described coupling comprises execution bundle adjustment;

1f) for each corresponding image of described more than first image, determine the first attitude data allowing to reach a conclusion to position and the direction about the reference frame relative to the corresponding image of shooting according to the result of described coupling;

1g) at least one image in described second image and described more than first image is compared;

1h) at least one rendering image in described more than first image, according to the result of described coupling and described first attitude data that is associated with at least one rendering image described, determine the covering position in described second image; And

1i) on the display device, at least one rendering image described on the fixed covering position be coated in described second image is shown.

2. method as claimed in claim 1, further comprising the steps of:

Several initial values of the first attitude data be associated with at least one image in described more than first image are provided;

Wherein, in step 1e) in, consider that at least one first attitude data described is to carry out described bundle adjustment.

3. method as claimed in claim 1, further comprising the steps of:

Determine the second attitude data allowing to reach a conclusion to position and the direction about the described reference frame relative to described second image of shooting according to the result of described comparison.

4. method as claimed in claim 1, further comprising the steps of:

The second attitude data allowing to reach a conclusion to position and the direction about the described reference frame relative to described second image of shooting is provided; And

Wherein, in step 1g) in, consider that described second attitude data is to carry out described comparison, and revise described second attitude data according to the result of described comparison.

5. method as claimed in claim 4, wherein, determines described covering position according to the second revised attitude data.

6. method as claimed in claim 1, wherein, in step 1d), 1e), 1f), 1g), 1h) at least one step perform in described second mobile device or at least one server.

7. method as claimed in claim 1, further comprising the steps of:

There is provided several reference-view of true environment to reference database and allow about the attitude data of reaching a conclusion relative to the position of reference global coordinates system and direction of being taken corresponding reference-view by video camera;

By comparing at least partially of the real object comprised at least one reference-view at least one real object shown in described second image and described reference-view, and the second attitude data of described image is mated with the attitude data of at least one reference-view described; And

Based at least one reference-view described attitude data revise described second attitude data at least partially at least partially, as the result of described coupling.

8. method as claimed in claim 1, wherein, step 1e) in described coupling comprise: the character pair of the characteristics of image of each image in described more than first image with other image in described more than first image is mated; And

Step 1g) in comprise described comparison: the character pair of the characteristics of image of at least one image in described more than first image with described second image is mated.

9. method as claimed in claim 1, wherein, described second mobile device comprises described display device or connects described display device.

10. method as claimed in claim 1, wherein, the coordinate system that described reference frame is geographical global coordinates system or is associated with the real object in described true environment.

11. methods as claimed in claim 1, wherein, at least one first mobile device described and described second mobile device are identical client devices, and at least one database wherein said comprises described client device or at least one server, or be connected to described client device or at least one server.

12. methods as claimed in claim 1, wherein, described first mobile device and described second mobile device are different client devices, and at least one database described comprises or is connected at least one server.

13. 1 kinds, for describing the method for virtual information in the view of true environment, comprise the following steps:

13a) by least one first image of at least one the first mobile device shooting true environment, and provide the first attitude data allowing to reach a conclusion to position and the direction about the reference frame relative to shooting at least one the first image described;

13b) by the correspondence location at least one first image described, add at least one virtual objects;

13c) according to the location of at least one virtual objects at least one first image described, second attitude data is provided, wherein, described second attitude data allows to reach a conclusion relative to the position of described reference frame and direction to about at least one virtual objects described;

13d) at least one virtual objects described, at least one first image described, described first attitude data and described second attitude data are stored at least one database;

The second image of described true environment 13e) is taken by the second mobile device;

13f) access store at least one database described described at least one virtual objects, at least one first image described, described first attitude data and described second attitude data;

Comparing at least partially 13g) by the real object comprised at least one real object shown at least one first image described and described second image;

13h) determine the 3rd attitude data according to the result compared, wherein, described 3rd attitude data allows to reach a conclusion to about relative to the position of reference frame of described second image of shooting and direction; And

13i) based on described second attitude data and described 3rd attitude data, display is superimposed with at least one virtual objects described of described second image on the display device.

14. as the method for claim 13, and wherein, described second mobile device comprises described display device or connects described display device.

15. as the method for claim 13, wherein, and the coordinate system that described reference frame is geographical global coordinates system or is associated with the real object in described true environment.

16. as the method for claim 13, wherein, at least one first mobile device described and described second mobile device are identical client devices, and at least one database wherein said comprises described client device or at least one server, or be connected to described client device or at least one server.

17. as the method for claim 13, and wherein, described first mobile device and described second mobile device are different client devices, and at least one database described comprises or is connected at least one server.

18. as the method for claim 13, wherein, step 13f), 13g) and 13h) at least one step perform in described second mobile device or at least one server.

19. as the method for claim 13, wherein, step 13g) in comprise described comparison: the character pair of the characteristics of image of at least one the first image described with described second image is mated.

20. 1 kinds, for describing the method for virtual information in the view of true environment, comprise the following steps:

20a) provide at least one first view of true environment by the first data glasses and allow the first attitude data to reaching a conclusion about the position of reference frame and the direction relative to the described first data glasses in location;

At least one first image of the described true environment of being taken by the video camera being connected to described first data glasses 20b) is provided;

20c) by location that is corresponding at least one first view of described true environment, add at least one virtual objects;

20d) according to the location of at least one virtual objects at least one first view of described true environment, second attitude data is provided, wherein, described second attitude data allows to reach a conclusion relative to the position of described reference frame and direction to about at least one virtual objects described;

20e) at least one virtual objects described, at least one first image described, described first attitude data and described second attitude data are stored at least one database;

At least one second view of true environment 20f) is provided by the second data glasses;

At least one second image of the described true environment of being taken by the video camera being connected to described second data glasses 20g) is provided;

20h) access store at least one database described described at least one virtual objects, at least one first image described, described first attitude data and described second attitude data;

Comparing at least partially 20i) by the real object comprised at least one real object shown at least one first image described and at least one second image described;

20j) determine the 3rd attitude data according to the result compared, wherein, described 3rd attitude data allows to reach a conclusion to about relative to the position of reference frame of the described second data glasses in location and direction; And

20k) based on described second attitude data and described 3rd attitude data, display is superimposed with at least one virtual objects described of at least one the second view described.

21., as the method for claim 20, wherein, determine described first attitude data according at least one first image described.

22. as the method for claim 20, and wherein, described first data glasses and described second data glasses are identical client device or different client devices.