CN115167675A - Augmented reality device - Google Patents

Abstract

The present disclosure provides an augmented reality device. Images of a real-world scene are obtained from one or more image sensors. The orientation and/or position of the image sensor is obtained from one or more state sensors. A real world object in the image of the real world scene is identified and data associated with the identified object is displayed on a viewing device, a predetermined pointing object performing a predetermined gesture on the real world object. The invention also provides a computer program comprising computer program code means for performing all the steps of the method of the invention when said program is run on a computer.

Description

Augmented reality device

This application is a divisional application for an invention patent application with the application number of 201280048836.8 (international application number PCT/IL2012/050376), the filing date of which is on September 19, 2012, and the invention title is "touch-free interface for augmented reality system".

technical field

The present invention relates to methods and systems for augmented reality.

Related Technology

References relevant to the context of the disclosed subject matter are listed below:

U.S. Patent No. 7,126,558;

US Published Patent Application 20110221669;

US Published Patent Application 20110270522;

GB2465280(A);

U.S. Published Patent Application 20120068913;

US Patent No. 7,215,322;

WO2005/091125;

WO 2010/086866

"Finger Tracking as an Input Device for AugmentedReality" by Crowley, J.L. et al. It was published in the June 1995 issue of Zurich's "International Workshop on Face and Gesture Recognition", Switzerland.

The identification of the above references should not be inferred to mean that these are in any way related to the patents of the disclosed subject matter.

Background technique

Augmented reality is the term for a real-time, direct or indirect view of a physical, real-world environment whose elements are augmented by computer-generated information, such as text, sound, video, graphics, or GPS data. Artificial information about the environment and its objects is thus overlaid on the real-world view or image. Augmentation typically occurs in real-time and in the semantic context of environmental factors, making information about the user's surrounding real world interactive and digitally actionable.

The main hardware components used for augmented reality are processors, displays, sensors, and input devices. These elements, in particular CPUs, displays, cameras and MEMS sensors such as accelerometers, GPS or solid state compasses, are present in portable devices such as smartphones, allowing them to act as augmented reality platforms.

Augmented reality systems have been widely used in entertainment, navigation, assembly processes, maintenance, medical procedures. Portable augmented reality systems have also been widely used in travel and tourism, where augmented reality is used to present information about real-world objects being viewed and location objects.

Use a head-mounted display, usually in the form of goggles or a helmet, to provide an immersive augmented reality experience. In the case of a head mounted display, virtual visual objects are superimposed on the user's view of the real world scene. Head-mounted displays are tracked with sensors that allow the system to align virtual information with the physical world. For example, tracking may be performed using any one or more of technologies such as digital cameras or other optical sensors, accelerometers, GPS, gyroscopes, solid state compasses, RFID, and wireless sensors. Head mounted displays are light see-through or video see-through. Light perspective uses solutions such as semi-silvered mirrors to pass the image through the lens and overlay the information to be reflected to the user's eyes, and transparent LCD projectors that display digital information and images directly or indirectly to the user retina.

SUMMARY OF THE INVENTION

The present invention provides an interactive system for augmented reality. The interactive system of the present invention includes a wearable data display device that may, for example, be incorporated into a pair of glasses or goggles. Wearable displays have devices (eg GPS) and compasses that provide location extraction capabilities. The system also includes a user interface that allows the user to select computer-generated data to enhance the real-world scene viewed by the user. The camera obtains an image of the real world scene being viewed. The processor detects predetermined objects, such as a user's finger, in the image of the real-world scene captured by the camera. When the user points to an element in the scene, data related to the element is displayed on the data display device and superimposed on the user's view of the scene.

Accordingly, in one aspect, the present invention provides a method for augmented reality comprising:

(a) obtaining images of real-world scenes from one or more image sensors;

(b) obtaining one or both of the orientation and position data of the image sensor from one or more state sensors;

(c) identifying a real-world object in the image of the real-world scene obtained by the one or more image sensors, a predetermined pointing object performs a predetermined gesture on the real-world object, and the gesture detection module utilizes the data provided by the one or more status sensors; and

(d) presenting data associated with the identified object on a display of the viewing device.

The image sensor may be selected from cameras, light sensors, IR sensors, ultrasonic sensors, proximity sensors, CMOS image sensors, short wave infrared (SWIR) image sensors or reflective sensors, IR sensors, ultrasonic sensors, proximity sensors, and reflective sensors. One or more of the status sensors may be selected from: optical sensors, accelerometers, GPS, gyroscopes, compass, magnetic sensors, sensors indicating the orientation of the device relative to the earth's magnetic field, gravity sensors, and RFID detectors.

The data associated with the identified object may be obtained by searching memory for data associated with the real world object.

The predetermined object may be, for example, a hand, a part of a hand, two hands, parts of two hands, a finger, a part of a finger, or a fingertip.

The viewing device may be configured to be worn by a user, eg, glasses or goggles. The viewing device may be incorporated into a mobile communication device.

The step of identifying in the image of the real-world scene obtained by the one or more image sensors may include: determining the position (X, Y) of the predetermined object in the image obtained by the image sensor; and determining one or both of the position and orientation of the display device provided by the sensor.

The method of the present invention may further include communicating with an external device or website. The communication may include sending a message to an application running on the external device, a service running on the external device, an operating system running on the external device, a service running on the external device A program, one or more application programs running on a processor of the external device, a software program running in the context of the external device, or one or more services running on the external device. The method may also include sending a message to an application running on the mobile communication device, a service running on the mobile communication device, an operating system running on the mobile communication device, an application running on the mobile communication device A program running on a communication device, one or more application programs running on a processor of the mobile communication device, a software program running in the context of the mobile communication device, or on the mobile communication device One or more services running.

The method may further include: from an application running on the external device, a service running on the external device, an operating system running on the external device, a program running on the external device, One or more application programs running on the processor of the external device, software programs running in the background of the external device, send messages requesting data related to real-world objects identified in the image , or send the message to one or more services running on the external device. The method may also include: from an application running on the mobile communication device, a service running on the mobile communication device, an operating system running on the mobile communication device, an operating system running on the mobile communication device A program running, one or more application programs running on a processor of the mobile communication device, a software program running in the background of the mobile communication device send a message requesting a data about real world objects, or send the message to one or more services running on the mobile communication device.

The message to the external device or website may be a command. The command may be selected from: a command to run an application on the external device or website, a command to stop an application running on the external device or website, a command to activate a service running on the external device or website A command, a command to stop a service running on the external device or website, or a command to send data related to a real-world object identified in an image.

The message to the mobile communication device may be a command. The command may be selected from: a command to run an application on the mobile communication device, a command to stop an application running on the mobile communication device or a website, a command to activate a service running on the mobile communication device , a command to stop a service running on the mobile communication device, or a command to send data related to real world objects identified in an image.

The method may further include: receiving data from the external device or website related to the real-world object identified in the image; and presenting the received data to a user.

Communication with the external device or website may take place through a communication network.

The command to the external device may be selected from: pressing a virtual key displayed on a display device of the external device; rotating a selection dial; switching desktops, running a predetermined software application on the external device; closing the external device Apps on the device; turn the speakers on or off; turn the volume up or down; lock the external device, unlock the external device, skip to another track on the media player or switch between IPTV channels; control the navigation application ; initiate a call, end a call, present notifications, display notifications; browse photo or music album galleries, scroll web pages, present emails, present one or more documents or maps, control in-game actions, point to maps, zoom in, or zooming out on a map or image, coloring on an image, grabbing and pulling an active icon from the display device, rotating an active icon, simulating touch commands on the external device, executing one or more multi-touch commands, touch gesture commands, typing, tap to show video to pause or play, mark or capture frames from video, present incoming messages; answer incoming calls, mute or reject incoming calls, turn on incoming call alerts; present notifications received from online community services; Present a notification generated by the external device, open a predetermined application, change the lock mode of the external device and open the recents application, change the lock mode of the external device and open an online service application or browser, change all change the lock mode of the external device and open an email application, change the lock mode of the external device and open an online service application or browser, change the lock mode of the device and open a calendar application, change the lock of the device mode and open a reminder application, change the lock mode of the device and open a predetermined application set by the user, set by the manufacturer of the external device or set by the service operator, activate the activation icon, select a menu item, on the display Move the pointer, manipulate the touch-free mouse on the monitor, activate icons, and change information on the monitor.

In the method of the present invention, the predetermined gesture can be selected from: page turning gesture, two-finger pinch motion, pointing, left-to-right gesture, right-to-left gesture, up gesture, down gesture, pressing gesture, open clenched fist, open clenched fist and move towards the image sensor, tap gesture, wave gesture, clapping gesture, reverse clapping gesture, hand in fist, pinch gesture, reverse pinch gesture, open Finger gesture, reverse finger open gesture, pointing at activation icon, holding activated object for predetermined amount of time, click activation icon, double click activation icon, right click activation icon, left click activation icon, bottom click activation icon, click on the activation icon from above, grab the activation icon as the object, point at the activation icon as the object from the right, point at the activation icon from the left, pass the activation icon from the left, push the object, clap, wave over the activation icon , perform an explode gesture, perform a tap gesture, perform a clockwise or counterclockwise gesture on the activation icon, swipe the icon, grab the activation icon with two fingers, and perform a tap-drag-release motion.

The data associated with the identified object may be any one or more of visual data, audio data, or textual data. The data associated with the identified object may be an activation icon. The activation icon may be a 2D or 3D activation icon. The activation icon is perceivable by the user in 3D space in front of the user.

The method of the present invention may have two or more modes of operation. The method may change the operating mode of the system after recognizing a predetermined gesture. The mode of operation may be specified by any one or more of: the gesture to be recognized, the algorithm active on the gesture detection module; the resolution of the image captured by the image sensor, and the image capture captured by the image sensor rate, the level of detail of the data to be presented, the activation icon to be presented to the user, the source of data to be presented, the level of detail of the data to be presented, the level of detail to be displayed on the display device activation icon, active online service.

The operating mode may be a mode selected from the group consisting of: a mode in which the image sensor video records an image after recognizing a predetermined gesture; a mode in which the microphone records sound after recognizing a predetermined gesture and stops recording after recognizing another predetermined gesture; continuous monitoring A mode of video or sound and after a predetermined gesture is detected, the video or sound is recorded starting a predetermined amount of time before the gesture is recognized and the recording is stopped after another predetermined gesture is recognized; mode for tagging live recorded video; mode for selecting an area in the field of view captured by the camera and copying the area to another location in the field of view and resizing it; using the selected area in the image tracker and present the pattern of the selection area in real time in the resize and relocation area on the display device; the pattern of capturing an image after a predetermined gesture is recognized.

The method of the present invention may further include running a tracking algorithm that tracks the identified real-world object and maintains the displayed associated visual data in a fixed position relative to the identified real-world object.

An object recognition module may be operable to detect the predetermined object only if the display device has a level of motion below a predetermined threshold.

The method may further include providing feedback when the predetermined gesture has been recognized. The feedback can be, for example, visual feedback, auditory feedback, haptic feedback, directional vibration, air haptic feedback, or ultrasonic feedback. The feedback may be a visual indication in a form selected from an activation icon displayed on the display device, a change in the activation icon displayed on the display device, a change in color of the activation icon displayed on the display device, Changes in the size of the activation icon displayed on the display device, animation of the activation icon displayed on the display device, indicator lights, indicators moving on the display device, all other images appearing on the display device, or A pointer appearing at the top of the video moving on the display device, and the appearance of the glow around the predetermined object. The feedback may be vibration, a directional vibration indication, or an air tactile indication.

In the method of the present invention, the portion of the activation icon displayed on the display device is not presented where the predetermined object is located, so that the predetermined object appears to be on top of the activation icon.

The activation icon may be removed from the display device when the display device has an activity level above a predetermined threshold. For example, the deleted icon on the display device may be deleted when the display device has an exercise level below the predetermined threshold.

The method may be brought into the active mode when a predetermined action is performed. The predetermined action may be selected from: when the user puts the predetermined object into a certain position or a ball, bringing the predetermined object into the field of view from below, for example, pointing at the said field of view of the camera. lower right corner or open hand in the camera field of view; when an activation icon is displayed and the user performs a predetermined gesture associated with the activation icon, such as pointing to the activation icon, a predetermined gesture is performed, such as from the right moving the hand left across the field of view, or performing a waving gesture at the location where the activation icon is presented, or by performing a gesture in the 3D space where the activation icon is perceived to be, by touching the device , or tap on the device if the device has an accelerometer to slide the floating activation icon from one position to another. As another example, if the device has a proximity sensor or an ultrasonic sensor, the system may enter the active mode when the user's hand approaches the device. The system may also be activated by a voice command, or when the user places the predetermined object in a specific location in the field of view. As another example, the system may enter the active mode only when there is relevant data associated with the real world in the user's field of view. At this point, the system can indicate to the user when there is relevant data to be presented, or when it is ready to interact.

The method of the present invention may further include attaching a visual indication to the real world object to indicate that there is a memory of data related to the real world object. The visual indication may be overlaid on the image of the real world object. The visual may be selected from activation icons, photographs, and images of envelopes.

The method of the present invention may further comprise a calibration process of recording one or more physical parameters of the predetermined object. The calibration process may include any one or more steps selected from: presenting activation icons on the display at different locations in 3D space; extracting physical properties of the predetermined object; and determining the predetermined object's physical properties. Correlation between size and its distance from the camera. The calibration process may include the steps of constructing a triangle whose apex is on one of the image sensors and in front of the predetermined object and whose sides are formed by the user's line of sight. The distance of the real world object to the camera may be estimated based on information extracted in the calibration.

The method may further include: displaying a keyboard capable of text input. The keyboard may be displayed upon detection of a predetermined gesture, such as a right-to-left gesture, presenting an open hand, presenting two open hands in a predetermined area of the field of view of the image sensor. The keyboard may be displayed after performing a tap gesture in a 3D typing area or sensing where a predetermined activation icon is located.

The present invention also provides a system comprising an apparatus configured to perform the method of the present invention.

The invention also provides a computer program comprising computer program code means for carrying out all the steps of the method of the invention when said program is run on a computer. The computer program may be embodied on a computer-readable medium.

Users can interact with visual images that are typically displayed through glasses. Thus, the user's view of reality is enhanced by the information presented on the display. One problem with augmented reality devices is the way the user interacts with and controls the device. Traditional control devices such as mice, trackballs or touchscreens are difficult to use with augmented reality devices. Using gesture recognition in augmented reality systems is not straightforward because the user is constantly moving in real time and therefore the augmented reality device is constantly moving in real time.

The present invention therefore provides a computer program product comprising instructions for causing a processor to perform a method comprising the steps of:

receive image information associated with the environment from an image sensor associated with the augmented reality device;

Display enhanced information about the environment on the display associated with the device;

Recognize device user gestures in image information;

associating gestures with augmented information; and

The displayed enhancement information is changed based on the association.

The enhanced information may include at least one of: information associated with objects in the environment; images associated with the environment; and distances associated with the environment.

The associating may include: determining a reference position in three-dimensional space of at least a portion of the user's hand; and determining at least one of enhancement information and image information data associated with the reference position.

The changing may include changing the enhancement information according to data associated with the reference location.

Description of drawings

In order to understand the invention and to understand how it may be implemented in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:

Figure 1 schematically shows a system for augmented reality according to an embodiment of the present invention;

Figure 2 illustrates a system for augmented reality including a set of goggles according to one embodiment of the present invention;

Figures 3a and 3b illustrate the system of Figure 2 in use;

Figure 4a shows a view of a real world scene displayed on a display device of the system of Figure 2; Figure 4b shows the view of Figure 4a with a user's finger pointing at an object in the view; and Figure 4c shows the view overlaid on Figure 4b visual text on the object related to the object the user's finger is pointing at;

Figures 5a, 5b illustrate a system for augmented reality integrated with a communication device according to another embodiment of the present invention; and

Fig. 6a shows designation of an area in the field of view of the image sensor by the user performing a gesture of "drawing" the outline of the area; Fig. 6b shows resizing of the selection area by performing a second gesture; Fig. 6c shows the resizing region; and Figure 6d shows the region after dragging to a new location in the field of view.

Detailed ways

Figure 1 schematically shows a system 30 for augmented reality according to one embodiment of the present invention. System 30 includes one or more image sensors 32 configured to obtain images of real world scenes. Any type of image sensor can be used in the system of the present invention, such as cameras, light sensors, IR sensors, ultrasonic sensors, proximity sensors, CMOS image sensors, short wave infrared (SWIR) image sensors, or reflective sensors.

The system 30 also includes a viewing device 34 having one or more display devices 35 that enable a user to view the real world scene and external information, such as images, video or audio signals, superimposed on the real world scene. Any type of display device that allows a user to see the real world scene and displayed data can be used in the system of the present invention.

Display device 35 may, for example, include a surface on which visual material is presented to the user or one or more projectors that display images directly to the user's retina. Processor 36 obtains orientation and/or position data of system 30 from one or more state sensors 38, which may be, for example, optical sensors, accelerometers, GPS, gyroscopes, solid state compasses, magnetic sensors, gravity sensors, and RFID detection any one or more of the devices. The processor 36 can be, for example, a special purpose processor, a general purpose processor, a DSP (Digital Signal Processor) processor, a GPU (Visual Processing Unit) processor, dedicated hardware, or a processor that can run on an external device. System 30 may run as software on viewing device 34 or another device 37 (eg, a smartphone) that incorporates other components of system 30 .

Processor 36 is configured to run gesture detection module 40 which identifies one or more real world objects to which predetermined objects are pointing in images of the real world scene obtained by image sensor 32 . Real world objects may be, for example, buildings or billboards. The determination of real world objects uses data provided by state sensors 38 . The predetermined object may be the user's finger or other object such as a stylus or a stick.

When the processor 36 has identified a real world object to which the predetermined object is pointing, the processor searches the memory 42 for data associated with the identified object. The data may be, for example, visual data, audio data, or textual data. The visual data may be textual information related to the recognized object. The processor then displays relevant visual data associated with the identified object on the display of the viewing device. Memory 42 may be integral with system 30, or may be located remotely and accessed through a communication network such as the Internet. The system 30 may thus include a communication module 39 that allows the system 30 to communicate with a network, a wireless network, a cellular network, an external device (eg, another device 30, a cell phone, a tablet), or an Internet website, or the like.

The data may be an activation icon. As used herein, the term "activation icon" represents an area in an image or video associated with one or more messages or commands activated by user interaction. The activation icon may be, for example, a 2D or 3D visual element, such as a virtual button, a virtual keyboard or an icon. The activation icon is activated by one or more predetermined objects, which are identifiable by the system and can be, for example, a stylus, a user's hand or part of a hand, one or more fingers, or a part of a finger such as a fingertip one or more of. Activation of one or more of the activation icons by a predetermined object generates a location to the operating system, one or more services, one or more applications, one or more devices, one or more remote applications, one or more remote applications Services, or messages or commands from one or more remote devices.

The processor 36 may be configured to send messages or commands to the device 37 or a remote device, applications running on the device, services running on the device 37, and operating systems running on the device, programs running on the device , a software program running in the background and one or more services running on the device, or a process running in the device. The message or command may be sent over a communication network such as the Internet or a cellular telephone network. The command may be, for example, a command to run an application on the device, a command to stop an application running on the device, a command to activate a service running on the device, a command to stop a service running on the device, or a command to communicate with the processor 36 Commands to send data about real world objects identified in the image to processor 36 .

The command may be a command to the device 37, such as pressing a virtual key displayed on the device's display device; rotating a selection dial; switching desktops, running predetermined software applications on the device; closing applications on the device; turning speakers on or off Turn up or down the volume; lock the device, unlock the device, skip to another track on the media player, or switch between IPTV channels; control the navigation application; initiate calls, end calls, show notifications, show notifications; browse photos or music album galleries, scroll web pages, render emails, render one or more documents or maps, control actions in games, control interactive video or animated content, edit videos or images, point to a map, zoom in or out on a map or image , colorize the image, pull the activation icon away from the display device, grab the activation icon and pull the activation icon from the display device, rotate the activation icon, simulate touch commands on the device, perform one or more multi-touch commands, touch gestures Command, type, tap to show video to pause or play, edit video or music commands, mark or capture frames from video, cut a subset of video from video, render incoming messages; answer incoming calls, mute or reject incoming calls, open incoming calls Reminders; present notifications received from web community services; present notifications generated by the device, change the lock mode of the device and activate the recent calls application, change the lock mode of the device and activate the online service application or browser, change the lock of the device mode and activate email application, change device lock mode and activate online service application or browser, change device lock mode and activate calendar application, change device lock mode and activate reminder application, change device lock mode and activate predetermined applications set by the user, set by the manufacturer of the device, or set by the service operator, activate the activation icon, select a menu item, move the pointer on the display, manipulate a touchless mouse, activate the activation icon on the display, and change information on the display.

The communication module can be used to send messages that can be located, for example, to a remote device. The message may be, for example, a command to a remote device. The command may be, for example, a command to run an application on the remote device, a command to stop an application running on the remote device, a command to activate a service running on the remote device, a command to stop a service running on the remote device. The message may be a command to the remote device selected from: pressing a virtual key displayed on the display device of the remote device; rotating the selection dial; switching desktops, running a predetermined software application on the remote device; closing the application on the remote device; Turn the speakers on or off; turn the volume up or down; lock the remote device, unlock the remote device, skip to another track on the media player, or switch between IPTV channels; control navigation applications; initiate calls, end calls, and present notifications , display notifications; browse photo or music album galleries, scroll web pages, present emails, present one or more documents or maps, control in-game actions, point to maps, zoom in or out on maps or images, colorize images, Grab the activation icon and pull the activation icon from the display device, rotate the activation icon, simulate a touch command on the remote device, perform one or more multi-touch commands, touch gesture commands, type, tap to display video to pause or play, mark frames or capture frames from video, present incoming messages; answer incoming calls, mute or reject incoming calls, turn on incoming call alerts; present notifications received from web community services; present notifications generated by remote devices, open scheduled applications, change remote devices change the lock mode of the remote device and open the recent call application, change the lock mode of the remote device and open the online service application or browser, change the lock mode of the remote device and open the email application, change the lock mode of the remote device and open the online service application Program or browser, changing the lock mode of the device and opening the Calendar application, changing the lock mode of the device and opening the Reminders application, changing the lock mode of the device and opening the user settings, the manufacturer settings of the remote device or the service operator settings , activate an icon, select a menu item, move a pointer on the display, manipulate a touchless mouse, activate an icon on the display, change information on the display.

The message may be a request for data associated with the identified object. The data request message may be located to an application, service, process, thread running on a device, or from an application, service, process or thread running on an external device, or an online service.

To reduce CPU resources, an object recognition module that detects a predetermined object may only be used when the headset is not significantly moving, as determined by the information obtained from the state sensor.

Figure 2 shows a system 2 for augmented reality according to one embodiment of the present invention. The system 2 includes a portable viewing device, which may be, for example, an interactive head-mounted eyepiece, such as a pair of glasses or goggles 4 . The goggles 4 have an image sensor 6 that obtains images of the real world scene 8 . The scene 8 may include, for example, one or more buildings 12 , or one or more billboards 14 . The goggles may have one or more display devices 10 positioned in the goggles 4 to be in front of the user's eyes when the goggles 4 are worn by the user. The display device 10 may be, for example, a see-through device, such as a transparent LCD screen, through which the real world scene is viewed and external data is presented. The system 2 also includes a processor 16 configured to identify predetermined objects in the images captured by the image sensor 6 that perform gestures or point to real-world objects in the real-world scene 8 or activation icons displayed to the user . System 2 also includes one or more position and/or orientation sensors 23, such as GPS, accelerometer, gyroscope, solid state compass, magnetic sensor, or gravity sensor.

Figures 5a, 5b illustrate a system 40 for augmented reality according to another embodiment of the present invention. The system 40 is integrated into a mobile communication device 42 such as a cell phone, tablet or camera. Figure 5a shows a front view of the communication device 42, while Figure 5b shows a rear view of the communication device 42. On the back of the communication device 42 is an image sensor 46 that obtains an image of the real world scene, the image sensor 46 being opposite the display device. The communication device 42 also has a display device 48 on the front face which is positioned in front of the user when the camera 46 is facing the real world scene. Display device 48 may be, for example, an LCD screen that presents to the user images of the real world scene obtained by camera 6 and visual data as explained below. The system 40 uses the camera 46, the display device 48, and the processor of the communication device 42, and also includes one or more status sensors contained in the housing of the communication device 42 not shown in Figures 5a, 5b middle. The processor is configured to identify predetermined objects in the images captured by the image sensor 46 that point to real-world objects in the real-world scene.

Figure 3a shows the system 2 in use. The goggles 4 are placed over the eyes of the user 18 . The user is oriented towards the real world scene 8 and thus views the scene 8 . Figure 3b shows the system 40 in use. The user 18 holds a communication device 42 having an image sensor 46 facing the real world scene 8 and a display device 48 facing the user.

System 2 or 40 now performs the following process. A view of the scene 8 that the user would see when using the system 2 or 40 is displayed on the display device. Figure 4a shows a view of the scene 8 that a user would see when viewing a real world scene 8 using the system 2 or 40. The processor 36 analyzes the images obtained by the image sensor to determine when a predetermined object in the image captured by the image sensor performs a predetermined gesture in relation to a real-world object in the real-world scene 8 .

The viewing device 34, such as the goggles 4 or the communication device 42, is not stationary in use due to the movement that occurs when the user walks or the movement of the user's head or hands. In this case, the signal generated by the sensor 38 may be noisy and inaccurate. In this case, the machine vision module 37 runs a tracking algorithm that tracks the identified real-world object and maintains the displayed relevant visual data in a fixed position relative to the identified real-world object.

The predetermined gesture related to the real world object or activation icon may, for example, point to the real world object or activation icon, or perform a page turning gesture on the real world object or activation icon. Activation icons may or may not be related to real-world objects.

Other possible predetermined gestures include page turning gestures, two-finger pinch movements (eg index finger and thumb or middle finger and thumb), pointing, left-to-right, right-to-left, up, down, press, Open clenched fist, open clenched fist and move towards image sensor, tap gesture, wave gesture, clap gesture, reverse clap gesture, hand in fist, pinch gesture, reverse pinch gesture, open fingers gesture, reverse open finger gesture, pointing at an activation icon or real-world object, pointing at an activation icon or real-world object for a predetermined amount of time, tapping an activation icon or real-world object, double-tapping an activation icon or real-world object, index finger tap Activate Icon or Real World Object, Middle Finger Click to Activate Icon or Real World Object, Bottom Click to Activate Icon or Real World Object, Top Click to Activate Icon, Grab Active Icon or Real World Object, Right Point to Activate Icon or Real World Object , Point to the activation icon or real-world object from the left, pass the activation icon or real-world object from the left, push the activation icon or real-world object, clap or wave over the activation icon or real-world object, perform an explode gesture, perform a tap gesture , perform a clockwise or counterclockwise gesture on the activation icon or real-world object, swipe the activation icon or real-world object, grab the activation icon or real-world object with two fingers, or perform a tap-drag-release motion.

The predetermined object may be, for example, the user's hand, a part of the user's hand, such as the user's finger 20, or parts of two different hands. Alternatively, the predetermined object may be a stylus or a stick.

When processor 16 determines that a predetermined gesture has been performed, this may be indicated to the user by any type of feedback, such as visual feedback, auditory feedback, haptic feedback, directional vibration, air haptic feedback, or ultrasonic feedback. The feedback may be a visual indication in a form selected from the group consisting of an activation icon displayed on the display device, a change in the activation icon displayed on the display device, a change in the color of the activation icon displayed on the display device, a change in the activation icon displayed on the display device. Changes in size, animation of activation icons displayed on display device, indicator lights, indicator moving on display device, vibration, directional vibration indication, air haptic indication. The indication may be provided by a pointer moving on the display device appearing on top of all other images or videos appearing on the display device. The visual feedback may be the appearance of a glow around the predetermined object when the system recognizes the predetermined object.

Gesture detection module 40 may use any method for detecting predetermined objects in images obtained by image sensor 32 . For example, the gesture detection module may detect predetermined objects as disclosed in WO2005/091125 or WO 2010/086866.

Processor 16 is also configured to determine real world objects in scene 8 on which predetermined gestures are performed. Thus, for example, in the image shown in Fig. 4b, the processor 16 would determine the position (X,Y) of the fingertip in the image and combine this information with the user's position from the state sensor 21 and the orientation of the goggles 4 It is determined that the user's finger 20 is pointing at the billboard 14 . Real world objects are thus recognized by the processor without presenting a cursor or other marker to the user to indicate the real world object the user wishes to select, thereby enabling direct pointing at the real world object to initiate interaction. The processor 16 searches a memory, which may be integral with the processor 16 or may be located remotely, for data related to the real-world object to which the user's finger 20 is pointing. For example, the memory may have stored data related to the billboard 14 . When the user points to an object in the scene 8 whose data is stored in memory or retrieved from a remote server such as a web site, the data is displayed on the display device 10 superimposed on a view of the user's scene. Thus, when the user points to the billboard 14 (Fig. 3a), visual data 21 related to the billboard 14 is displayed on the display device 10, as shown in Fig. 4c.

Visual data 21 may be static or dynamic. The visual data 21 may include one or more activation icons such that when a predetermined gesture is performed relative to one of the activation icons, the command associated with the activation icon is executed. The command may, for example, be to display specific visual material related to the selected real-world object. The activation icon may be a 2D or 3D activation icon and may be presented to the user such that the user perceives the icon in 3D space in front of him. As used herein, an activation icon is an area in a 2D or 3D image or video associated with one or more messages activated by user interaction. The activation icon may be, for example, a 2D or 3D visual element. The activation icon can be a virtual button, a virtual keyboard, a 2D or 3D activation icon, an area in an image or video. An activation icon may contain two or more activation icons.

The processor may not render the portion of the activation icon where the predetermined object is located, so that the predetermined object appears to be on top of the activation icon. When the user moves his head quickly, the activation icon can be deleted, and then when the head movement is lower than the predetermined movement speed, the activation icon is returned.

System 2 may have two or more modes of operation, and processor 16 may be configured to recognize one or more predetermined gestures to change between modes of operation. Thus, gestures can be used to turn the system on or off, select a source of visual material to be presented, select the level of detail of visual material to be presented, select a button or activate an icon to be presented to the user, or activate an online service, such as a real world on selection Object's online service. Another mode of operation may be to start video recording of images with an image sensor and/or to record sounds with a microphone after a predetermined gesture is recognized, and stop recording after another predetermined gesture is recognized. Another mode of operation is to continuously monitor video and/or sound, but after a predetermined gesture is detected, record video/sound starting a predetermined amount of time before the gesture is recognized, and stop recording after another predetermined gesture is recognized. The predetermined time can be defined by the user. Another mode of operation is to add tags to captured and real-time recorded video after a predetermined gesture is recognized.

Figures 6a to 6d illustrate another mode of operation. In Fig. 6a, the area 62 in the field of view 60 captured by the image sensor is designated by the user performing a gesture of "drawing" the outline of the area, as shown in dashed lines in Figs. 6a-6d. The selection area is then resized by the user performing a second gesture, such as separating two fingers or bringing the two fingers closer together as indicated by arrow 66 in Figure 6b, until the selection area reaches the desired size (Figure 6c). 67 in ). Region 67 is then dragged to a new position in the field of view (Fig. 6d) and copied to the new position in the field of view. The system then uses the tracker on the selection area, and the selection area is rendered in real-time in the user-set resize and reposition area on the display device.

To minimize CPU resources, for each displayed active icon, the image area containing the displayed active icon bounding box around the displayed active icon may be defined to remain unchanged. The system uses a machine vision tracker to track this bounding box. The distance between the positions of the bounding boxes in two frames of the video sequence is less than a predetermined distance determined using the video tracker, and the correlation value of the bounding box's tracker is below the predetermined value.

When the system is in a mode of operation in which only activation icons can be activated and no real world objects can be activated, the CPU can be minimized by only searching for predetermined objects in the vicinity of each displayed activation icon. To further reduce the CPU, the object recognition module is only activated when it is determined that the headset has not moved significantly, as obtained from the state sensor.

Users can select different filters to filter data related to real-world objects, such as "show data generated only by friends," or filters that show data from registered sources or data generated in the past three months.

System 2 may have a standby mode in which system 2 consumes minimal power. For example, the active mode may differ from the standby mode in the number of video frames per second analyzed by the system, the image resolution analyzed, the fraction of image frames analyzed, and/or the active detection module. System 2 can be brought into active mode by any technique. For example, the system 2 may be brought into active mode by the action of bringing a predetermined object into the field of view from below when the user places it in a position or mass, such as pointing to the lower right corner of the camera's field of view or at Open the hand in the camera's field of view; when the activation icon is displayed and the user performs a predetermined gesture associated with the activation icon, such as pointing at the activation icon, performs a predetermined gesture, such as moving the hand across the field of view from right to left, or Perform a wave gesture where the activation icon is presented, or by performing a gesture in 3D space where the activation icon is perceived, by touching the device, or tapping on the device if the device has an accelerometer, sliding the floating activation icon from one position to another location. As another example, if the device has a proximity sensor or an ultrasonic sensor, the system may enter an active mode when the user's hand is close to the device. The system can also be activated by a voice command, or when the user places a predetermined object in a specific location in the field of view. As another example, the system may enter active mode only when there is relevant data associated with the real world in the user's field of view. At this point, the system can indicate to the user when there is relevant data to be presented, or when it is ready to interact.

A visual indication can be attached to the real world object to let the user know that there is data related to the real world object.

Indications of relevant data may be overlaid on the location of real world objects, as a small visual indication such as an activation icon "i" may indicate information, while a sign for "photos" may indicate images related to real world objects, or an "envelope" The flag indicates messages left by friends or other users related to real-world objects. The data may be presented when the user performs a predetermined gesture associated with activating the icon.

The system 2 may be configured to record various physical parameters of the predetermined object through a calibration process, thereby facilitating the processor 2 to identify the predetermined object in the images obtained by the camera. This may be done, for example, by presenting the activation icon to the user at different locations in 3D space on the display; extracting physical properties of the predetermined object, such as the size or orientation of the predetermined object; and determining the size of the predetermined object and its relationship to the camera the correlation between the distances. Calibration may include calculating the camera's triangulation, the user's line of sight, and the front end of a predetermined object to determine the user's pointing. Improves accuracy by estimating the distance of real-world objects from the camera based on information extracted from calibration.

The processor may be configured to identify images of the real world scene obtained at the camera by a user of the system of the present invention. The identification of another user in a real-world scenario can be performed, for example, by notifying a remote server of the location of the device in a particular geographic area. The location of other devices can be sent to all devices in the geographic area.

When there is a communication link between the two systems of the present invention, the two systems can be used to play games. The other user may be represented as a computer avatar, with whom the user may interact through gestures, such as sending a message such as "like" to the other user.

The processor may be configured to display a keyboard capable of text input using one or more fingers or hands. Display of the keyboard may begin after detection of a predetermined gesture, such as a right-to-left gesture, or presenting an open hand, or presenting two open hands in a predetermined area of the camera's field of view, so The predetermined area is, for example, the bottom of the field of view. Another way to start showing the keyboard is when the user performs a tap gesture in the typing area or in the 3D space where the sensory active icon is located. The keyboard can be used, for example, to write a note, conduct a search, or communicate with online services such as Skype or twitter by typing on a virtual keyboard. The system may not render the portion of the keyboard where the predetermined object is located so that the predetermined object appears to be on top of the keyboard to create the illusion that the predetermined object, eg, the user's hand, appears to be "above" the keyboard.

When the system is in input mode, an animated hand can be presented on the keyboard with a position relative to the user's hand and fingers. The fingertips of the animated hand can be positioned above the virtual keys where the key characters are seen. The keyboard and animated hands are preferably opaque so that the user cannot see the background behind the keyboard. This tends to make the keyboard clearer to the user.

Claims (10)

1. An augmented reality device comprising: at least one processor configured to: receiving video frames of image information associated with a real-world scene from an image sensor; detecting a predetermined gesture performed by the user in the image information; specifying a region of selected image information associated with a real-world object at a first location in the video frame, wherein the selected image information is associated with a real-world scene that does not include a user's hand, the specified region is associated with detection is associated with the predetermined gesture received; copying the designated area of the selected image information to a second location in the user's field of view; and Selecting image information associated with the real-world scene is tracked to present a real-time view of the real-world object in the selected information at the second location. 2. The augmented reality device of claim 1, wherein the predetermined gesture includes drawing an outline of the designated area. 3. The augmented reality device of claim 1, wherein the at least one processor is further configured to resize the designated area. 4. The augmented reality device of claim 3, wherein the second predetermined gesture comprises separating two fingers, or bringing two fingers closer together. 5. The augmented reality device of claim 1, wherein the at least one processor is further configured to zoom in or out in association with a second predetermined gesture detected in the image information. 6. The augmented reality device of claim 3, wherein the at least one processor is further configured to zoom in or out in association with resizing of the designated area. 7. The augmented reality device of claim 1, wherein the at least one processor is further configured to capture from the video frame of image information related to a second predetermined gesture detected in the image information linked frame. 8. The augmented reality device of claim 1, wherein the second location is defined by a user. 9. The augmented reality device of claim 8, wherein the second location is defined by a user performing a drag gesture associated with the designated area. 10. The augmented reality device of claim 1, wherein the at least one processor is further configured to employ a tracker on selected image information to resize a region in the user's field of view presents a real-time view of the real-world objects in the selected information.

Images