CN110419018B - Automatic control of wearable display devices based on external conditions - Google Patents

Abstract

Embodiments of the wearable device may include a Head Mounted Display (HMD) that can be configured to display virtual content. When a user interacts with visual or audible virtual content, the wearable user may encounter a triggering event, such as an emergency or unsafe condition, detection of one or more triggering objects in the environment, or determining characteristics of the user's environment (e.g., home or office). Embodiments of the wearable device may automatically detect the trigger event and automatically control the HMD to de-emphasize, block, or stop displaying virtual content. The HMD may include buttons that may be actuated by a user to manually de-emphasize, block, or stop displaying virtual content.

Description

Automatic control of wearable display devices based on external conditions

Cross References to Related Applications

This application claims a patent application entitled "MANUAL ORAUTOMATIC CONTROL OF WEARABLE DISPLAY DEVICE BASED ON EXTERNAL CONDITIONS" filed on December 29, 2016 pursuant to 35 U.S.C. §119(e). priority of U.S. Provisional Application No. 62/440099, the disclosure of which is incorporated herein by reference in its entirety.

technical field

The present disclosure relates to mixed reality imaging and visualization systems, and more particularly to automatic control of mixed reality imaging and visualization systems based on external conditions.

Background technique

Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality" or "mixed reality" experiences, in which images, or parts thereof, are digitally reproduced in such a way that they appear or may be perceived as real presented to the user. Virtual reality or "VR" scenarios typically involve the presentation of digital or virtual image information opaque to otherwise actual real-world visual input; augmented reality or "AR" scenarios typically involve the presentation of digital or virtual image information as a reflection of the real world around the user. Augmentation of visualization; Mixed reality or "MR" involves merging the real and virtual worlds to produce new environments where physical and virtual objects coexist and interact in real time. The human visual perception system has proven to be very complex, and it is challenging to produce AR technologies that facilitate a comfortable, natural, rich presentation of virtual image elements along with other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with AR and VR technologies.

Contents of the invention

Embodiments of a wearable device may include a head-mounted display (HMD) that may be configured to display virtual content. When the user interacts with visual or auditory virtual content, the user of the wearable device may encounter a trigger event, such as an emergency or unsafe situation, detect one or more trigger objects in the environment, or detect that the user has entered a specific environment (for example, home or office). Embodiments of the wearable device may automatically detect triggering events and automatically control the HMD to de-emphasize, block, or stop displaying virtual content. The HMD can include buttons that can be actuated by the user to manually de-emphasize, prevent, or stop displaying virtual content. In some implementations, the wearable device can restart or resume the virtual content in response to detecting a termination condition.

The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will be apparent from the description, drawings, and claims. Neither this Summary nor the Detailed Description that follows is intended to define or limit the scope of the inventive subject matter.

Description of drawings

FIG. 1A depicts an illustration of a mixed reality scene with some virtual reality objects and some physical objects viewed by a person.

Figure IB illustrates the field of view and gaze of a wearer for a wearable display system.

Fig. 2 schematically shows an example of a wearable display system.

Fig. 3 schematically illustrates aspects of a method of simulating a three-dimensional image using multiple depth planes.

Figure 4 schematically shows an example of a waveguide stack for outputting image information to a user.

Figure 5 shows an exemplary exit beam that may be output by the waveguide.

Figure 6 is a schematic diagram illustrating an optical system including a waveguide, an optical coupler subsystem for optically coupling light to or from the waveguide, and for producing a multifocal volumetric display , image or light field control subsystem.

7 is a block diagram of an example of a wearable system.

8 is a process flow diagram of an example of a method of presenting virtual content related to a recognized object.

9 is a block diagram of another example of a wearable system.

10 shows a schematic diagram of an example of various components of a wearable system including the environment and sensors.

11A and 11B illustrate an example of silencing a head mounted display (HMD) in a surgical setting.

Figure 11C shows an example of silencing an HMD in an industrial context.

Figure 1 ID shows an example of silencing an HMD in an educational context.

Figure 1 IE shows an example of silencing an HMD in the context of shopping.

FIG. 11F shows an example of selectively blocking virtual content in a work environment.

FIG. 11G illustrates an example of selectively blocking virtual content in a lounge environment.

12A, 12B, and 12C illustrate examples of silencing virtual content presented by an HMD based on a triggering event.

FIG. 12D illustrates an example of silencing virtual content when a change in the user's environment is detected.

13A and 13B illustrate an example process for silencing an augmented reality display device based on a trigger event.

13C illustrates an example flow diagram for selectively blocking virtual content in an environment.

FIG. 14A shows a warning message that may be displayed by the HMD in response to manual actuation of a real button.

14B is a flowchart illustrating an example process for manually actuating the HMD's silent mode of operation.

Throughout the drawings, reference numerals may be re-used to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and not to limit the scope of the present disclosure.

Detailed ways

overview

The display system of the wearable device may be configured to present virtual content in an AR/VR/MR environment. Virtual content may include visual and/or audio content. While using a head-mounted display device (HMD), a user may encounter situations where it may be desirable to de-emphasize or not provide some or all of the virtual content. For example, a user may encounter an emergency or unsafe situation during a time when the user's full attention should be on the actual physical reality without potential distraction from virtual content. In such a situation, presenting virtual content to the user may cause perceptual confusion when the user is trying to deal with the actual physical content of the real world and the virtual content provided by the HMD. Accordingly, embodiments of the HMD may provide manual or automatic control of the HMD in situations where it may be desirable to de-emphasize or cease display of virtual content, as described further below.

Furthermore, while wearable devices can present a large amount of information to a user, in some cases it can be difficult for the user to sift through virtual content to identify content that the user is interested in interacting with. Advantageously, in some embodiments, the wearable device can automatically detect the user's location and selectively block (or selectively allow) virtual content based on that location, so the wearable device can appear more relevant to the user and Virtual content appropriate to the user's environment (eg, location) (eg, whether the user is at home or work). For example, wearable devices can present various virtual content related to video games, scheduled conference calls, or work emails. If the user is in an office, the user may need to watch work-related virtual content, such as conference calls and emails, while video game-related virtual content is blocked so that the user can focus on work.

In some implementations, the wearable device can automatically detect changes in the user's location based on image data acquired by an outward-facing imaging system (alone or in combination with a position sensor). The wearable device may automatically apply settings appropriate for the current location in response to detecting that the user has moved from one environment to another. In some implementations, the wearable system can silence virtual content based on the user's environment (also referred to as a scene). For example, a living room in a home and a mall could both be considered entertainment scenarios, so similar virtual content could be blocked (or allowed) in both environments. Virtual content may also be blocked (or allowed) based on whether content with similar characteristics is blocked (or allowed). For example, a user may choose to block social networking applications in an office environment (or may choose to allow only work-related content). Based on this configuration provided by the user, for an office environment, the wearable system can automatically block video games because both video games and social networking applications have entertainment properties.

Although examples are described with reference to silencing virtual content, similar techniques may also be applied to silence one or more components of a wearable system. For example, a wearable system may silence an inward-facing imaging system to preserve the system's hardware resources in response to an emergency situation (eg, fire). Furthermore, although some examples are described as selectively blocking certain virtual content in certain environments, this is for example only and a mixed reality installation may additionally or alternatively selectively allow different virtual content, to achieve essentially the same result as blocking.

Example of 3D display for wearable systems

Wearable systems (also referred to herein as augmented reality (AR) systems) can be configured to present 2D or 3D virtual images to a user. An image may be a still image, a frame of a video or a video, a combination thereof, and the like. At least a portion of the wearable system can be implemented on a wearable device that can, alone or in combination, present a VR, AR, or MR environment for user interaction. A wearable device can be used interchangeably as an AR device (ARD). Additionally, the term "AR" may be used interchangeably with the term "MR" for the purposes of this disclosure.

FIG. 1A depicts an example of a mixed reality scene with some virtual reality objects and some physical objects viewed by a person. In FIG. 1A , an MR scene 100 is depicted in which a user of MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120 . In addition to these items, the user of the MR technology also perceives that he "sees" a robot statue 130 standing on a real-world platform, and a flying cartoon-like avatar character 140 that appears to be an avatar of a bumblebee , even if these elements do not exist in the real world.

In order for a 3D display to produce a realistic perception of depth, and more specifically, a simulated surface perception of depth, it may be desirable for each point in the display's field of view to produce an accommodative response corresponding to its virtual depth. If the accommodative response to a displayed point does not correspond to the virtual depth at that point, as determined by convergence and binocular depth cues for stereopsis, the human eye may experience an accommodation conflict, resulting in erratic imaging, unwanted eye fatigue, Headaches, and the almost complete lack of surface depth in the absence of regulatory information.

FIG. 1B shows a person's field of view (FOV) and field of gaze (FOR). FOV includes the portion of the user's environment that the user perceives at a given time. This field of view can change as the person moves, moves their head, or moves their eyes or gazes.

FOR includes a portion of the user's surroundings that can be sensed by the user via the wearable system. Thus, for a user wearing a head-mounted display device, the field of gaze may include substantially the entire 4π spherical solid angle around the wearer, since the wearer can move his or her body, head, or eyes to perceive substantially the entire 4π spherical solid angle in space. any direction. In other contexts, the user's movement may be more constrained, so the user's field of gaze may subtend a smaller solid angle. FIG. 1B shows such a field of view 155 including a central region and a peripheral region. The central field of view will provide a person with a corresponding view of objects in the central area of the environment view. Similarly, the peripheral field of view will provide a person with a corresponding view of objects in the surrounding area of the environment view. In this case, what is considered central and what is considered peripheral is a function of the direction the person is looking in and thus their field of view. Field of view 155 may include objects 121 , 122 . In this example, central field of view 145 includes object 121 while another object 122 is in the peripheral field of view.

Field of view (FOV) 155 may contain multiple objects (eg, objects 121, 122). The field of view 155 may depend on the size or optical characteristics of the AR system, for example, the transparent aperture size of a transparent window or the lens of a head-mounted display through which light passes from the real world in front of the user to the user's eyes. In some embodiments, as the posture of user 210 (eg, head posture, body posture, and/or eye posture) changes, field of view 155 may change accordingly, and objects within field of view 155 may also change. As described herein, a wearable system may include sensors such as a camera that monitors or images objects in field of gaze 165 as well as objects in field of view 155 . In some such embodiments, the wearable system may alert the user to unnoticed objects or events occurring within the user's field of view 155 and/or occurring outside the user's field of view but within the gaze field 165 . In some embodiments, the wearable system may also distinguish between content that directs the user's 210 attention or that does not.

Objects in FOV or FOR can be virtual objects or physical objects. Virtual objects may include, for example, operating system objects such as a terminal for entering commands, a file manager for accessing files or directories, icons, menus, applications for audio or video streaming, notifications from the operating system, and the like. Virtual objects may also include objects in applications (eg, avatars), virtual objects in game graphics or images, and the like. Some virtual objects can be operating system objects and objects in applications. Wearable systems can add virtual elements to existing physical objects viewed through the transparent optics of a head-mounted display, allowing users to interact with physical objects. For example, a wearable system could add a virtual menu associated with a medical monitor in the room, where the virtual menu could provide the user with options to turn on or adjust medical imaging equipment or dose controls. Thus, the HMD can present additional virtual image content to the wearer in addition to objects in the user's environment.

FIG. 1B also shows a field of gaze (FOR) 165 that includes a portion of the environment around person 210 that person 210 can perceive, for example, by turning their head or redirecting their gaze. The central portion of the field of view 155 of the eye of a person 210 may be referred to as the central field of view 145 . The area within field of view 155 but outside central field of view 145 may be referred to as the peripheral field of view. In FIG. 1B , gaze field 165 may contain a set of objects (eg, objects 121 , 122 , 127 ) that can be perceived by a user wearing the wearable system.

In some embodiments, the object 129 may be outside the user's visual FOR, but may still potentially be sensed (depending on their position and field of view) by sensors (e.g., cameras) on the wearable device and displayed for the user 210 or otherwise used by the wearable device associated with object 129. For example, object 129 may be located behind a wall in the user's environment such that the user cannot visually perceive object 129 . However, a wearable device may include sensors (such as radio frequency, Bluetooth, wireless, or other types of sensors) capable of communicating with object 129 .

Example showing the system

VR, AR and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are presented to the viewer. The image can be different for each depth plane (e.g., providing a slightly different rendering of the scene or object), and the image can be individually focused by the viewer's eyes, helping to be focused at different depth planes as needed Depth cues are provided to the user based on accommodation of the eye for different image features of the scene above or based on viewing different image features at different depth planes out of focus. As discussed elsewhere in this paper, such depth cues provide reliable depth perception.

FIG. 2 shows an example of a wearable system 200 that may be configured to provide AR/VR/MR scenarios. The wearable system 200 may also be referred to as an AR system 200 . The wearable system 200 includes a display 220 , and various mechanical and electronic modules and systems that support the functionality of the display 220 . Display 220 may be coupled to frame 230 , which may be worn by a user, wearer, or viewer 210 . Display 220 may be positioned in front of user 210's eyes. The display 220 may present AR/VR/MR content to the user. The display 220 may include a head mounted display (HMD) worn on the user's head. In some embodiments, speaker 240 is coupled to frame 230 and positioned near the user's ear canal (in some embodiments, another speaker (not shown) is positioned near the user's other ear canal to provide stereo/conformable audio) voice control). Wearable system 220 may include an audio sensor 232 (eg, a microphone) for detecting audio streams from the environment and capturing ambient sounds. In some embodiments, one or more other audio transducers (not shown) are positioned to provide stereo reception. Stereo reception can be used to locate sound sources. Wearable system 200 may perform speech or speech recognition on the audio stream.

Wearable system 200 may include an outward-facing imaging system 464 (shown in FIG. 4 ) that views the world in the environment around the user. The wearable system 200 may also include an inward-facing imaging system 462 (shown in FIG. 4 ) capable of tracking the user's eye movements. The inward-facing imaging system can track the movement of one eye or the movement of both eyes. Inwardly facing imaging system 462 may be attached to frame 230 and may be in electrical communication with processing module 260 or 270, which may process image information acquired by inwardly facing imaging system to determine, for example, the pupil diameter or orientation of the eye, the user's The eye movement or eye gesture of 210 .

As an example, wearable system 200 may use outward-facing imaging system 464 or inward-facing imaging system 462 to acquire images of the user's gestures. An image may be a still image, a frame of a video or a video, a combination thereof, and the like.

Wearable system 200 may include a user-selectable reality button 263 that may be used to attenuate visual or auditory content presented by wearable system 200 to the user. When the reality button 263 is actuated, the visual or auditory virtual content is reduced (compared to normal display conditions) so that the user perceives more of the actual physical reality that occurs in the user's environment. The reality button 263 may be touch or pressure sensitive and may be provided on the frame 230 of the wearable system 200 or on the battery power pack (eg, worn near the user's waist, eg, on a belt clip). Reality button 263 will be further described below with reference to FIGS. 14A and 14B.

Display 220 may be operatively coupled (such as by wired leads or a wireless connection) to local data processing module 250, which may be mounted in various configurations, such as fixedly attached to frame 230, fixedly attached to To a helmet or hat worn by the user, embedded within a headset, or otherwise detachably attached to the user 210 (eg, in a backpack configuration, in a strap-coupled configuration).

Local processing and data module 260 may include hardware processors and digital storage such as non-volatile memory (eg, flash memory), both of which may be used to assist in processing, caching and storing data. This data may include: a) data captured from sensors (which may, for example, be operatively coupled to frame 230 or otherwise operably attached to user 210), such as image capture devices (e.g., inwardly facing cameras in imaging systems or outward-facing imaging systems), audio sensors (eg, microphones), inertial measurement units (IMUs), accelerometers, compasses, global positioning system (GPS) units, radios, or gyroscopes; or b) Data acquired and/or processed using remote processing module 270 or remote data repository 280 may be communicated to display 220 after such processing or retrieval. Local processing and data module 260 may be operably coupled to remote processing module 270 or remote data repository 280 via communication link 262 or 264, such as via a wired or wireless communication link, such that these remote modules may serve as local processing and data Module 260 resources. Additionally, remote processing module 270 and remote data repository 280 may be operatively coupled to each other.

In some embodiments, remote processing module 270 may include one or more processors configured to analyze and process data or image information. In some embodiments, remote data repository 280 may include a digital data storage facility that may be available through the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in local processing and data modules, allowing completely autonomous use from remote modules.

Example Environmental Sensors

Environmental sensors 267 may be configured to detect objects, stimuli, people, animals, locations, or other aspects of the world around the user. As further described with reference to FIGS. 11A-11C , information obtained by environmental sensors 267 may be used to determine one or more triggering events that enable the wearable device to silence audio or virtual perception. Environmental sensors may include image capture devices (e.g., cameras, inward-facing imaging systems, outward-facing imaging systems, etc.), microphones, inertial measurement units (IMUs), accelerometers, compasses, global positioning system (GPS) units, radios , gyroscope, altimeter, barometer, chemical sensor, humidity sensor, temperature sensor, external microphone, light sensor (eg, photometer), timing device (eg, clock or calendar), or any combination or subcombination thereof. In some embodiments, environmental sensors may also include various physiological sensors. These sensors can measure or evaluate the user's physiological parameters, such as heart rate, respiration rate, galvanic skin response, blood pressure, EEG status, etc. The environmental sensor may also include a transmitting device configured to receive signals such as laser light, visible light, invisible light wavelengths, or sound (eg, audible sound, ultrasound, or other frequencies). In some embodiments, one or more environmental sensors (eg, cameras or light sensors) may be configured to measure ambient light (eg, brightness) of the environment (eg, to capture lighting conditions of the environment). Physical contact sensors such as strain gauges, curb detectors, etc. may also be included as environmental sensors. Additional details regarding environmental sensor 267 are further described with reference to FIG. 10 .

Local processing and data module 260 may be operatively coupled to remote processing module 270 and/or remote data repository 280 via communication links 262 and/or 264 (e.g., via wired or wireless communication links), such that these remote modules may be used with Source for local processing and data module 260. Additionally, a remote processing module and a remote data repository can be operably coupled to one another.

Wearable system 200 may also be configured to receive other environmental inputs, such as Global Positioning Satellite (GPS) location data, weather data, date and time, or other information that may be received from the Internet, satellite communications, or other suitable wired or wireless data communication methods. available environmental data. The processing module 260 may be configured to access further information characterizing the user's location, such as pollen counts, demographics, air pollution, environmental toxins, information from smart thermostats, lifestyle statistics, or relationships with other users, buildings, or healthcare providers. the proximity. In some embodiments, the information characterizing the location may be accessed using a cloud-based or other remote database. Local processing module 270 may be configured to obtain such data and/or further analyze data from any one or combination of environmental sensors.

Example of 3D light field display

The human visual system is complex and providing a realistic perception of depth is challenging. Without being bound by theory, it is believed that a viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. The vergent motion of the two eyes relative to each other (eg, the rotational motion of the pupils toward or away from each other to focus the eyes' line of sight on an object) is closely related to the focusing (or "accommodation") of the lenses of the eyes. Under normal circumstances, changing the focus of the lens of the eye, or causing the eye to accommodate to change the focus from one object to another at a different distance, would be referred to as the accommodation-vergence reflex. )” relationship automatically results in a matching change of vergence to the same distance. Also, under normal conditions, a change in vergence will trigger a matching change in accommodation. A display system that provides a better match between accommodation and vergence can result in a more realistic or comfortable three-dimensional image simulation.

FIG. 3 illustrates aspects illustrating a method of simulating a three-dimensional image using multiple depth planes. Referring to FIG. 3, objects at different distances on the z-axis from the eyes 302 and 304 are accommodated by the eyes 302 and 304 so that those objects are in focus. Eyes 302 and 304 exhibit specific accommodation states to bring objects at different distances along the z-axis into focus. Thus, it can be said that a particular accommodation state is associated with a particular one of the depth planes 306 that has an associated focal length such that when the eye is in the accommodation state for that depth plane, objects in the particular depth plane or part of the subject in focus. In some embodiments, three-dimensional images can be simulated by providing different presentations of images to each of eyes 302 and 304, and three-dimensional images can also be simulated by providing different presentations of images corresponding to each of the depth planes image. Although shown separated for clarity of illustration, it is understood that the fields of view of eyes 302 and 304 may overlap, for example, with increasing distance along the z-axis. Additionally, while shown as flat for ease of illustration, it is understood that the profile of the depth plane may be curved in physical space such that all features in the depth plane are in focus with the eye at a particular accommodation state. Without being bound by theory, it is believed that the human eye can generally interpret a limited number of depth planes to provide depth perception. Thus, by providing the eye with a different presentation of the image corresponding to each of these limited number of depth planes, a highly confident simulation of perceived depth can be achieved.

waveguide stack assembly

Figure 4 shows an example showing a waveguide stack for outputting image information to a user. Wearable system 400 includes a waveguide stack or stacked waveguide assembly 480 that can be used to provide three-dimensional perception to the eye/brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, the wearable system 400 may correspond to the wearable system 200 of FIG. 2 , wherein FIG. 4 schematically shows some parts of the wearable system 200 in more detail. For example, in some embodiments, waveguide assembly 480 may be integrated into display 220 of FIG. 2 .

With continued reference to FIG. 4 , the waveguide assembly 480 may further include a plurality of features 458 , 456 , 454 , 452 positioned between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. However, they may simply be spacers (eg cladding or structures for forming air gaps).

The waveguides 432b, 434b, 436b, 438b, 440b or the plurality of lenses 458, 456, 454, 452 may be configured to send image information to the eye with various levels of wavefront curvature or light divergence. Each waveguide level can be associated with a particular depth plane and can be configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be used to inject image information into waveguides 440b, 438b, 436b, 434b, 432b, each of which may be configured to distribute incident light through each respective A waveguide for output to the eye 410 . Light exits the output surfaces of the image injection devices 420, 422, 424, 426, 428 and is injected into respective input edges of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam (e.g., a collimated beam) can be injected into each waveguide to output clones oriented toward the eye 410 at a particular angle (and amount of divergence) corresponding to the depth plane associated with the particular waveguide. The entire field of view of the collimated beam.

In some embodiments, the image injection devices 420, 422, 424, 426, 428 are discrete displays, each display generating image information for injection into a respective waveguide 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, the image injection device 420, 422, 424, 426, 428 is the output of a single multiplexed display, which may be fed to the image injection device 420, for example, via one or more light guides, such as fiber optic cables. , 422, 424, 426, 428, each of the image injection devices uses a tube to deliver image information.

Controller 460 controls the operation of stacked waveguide assembly 480 and image injection devices 420 , 422 , 424 , 426 , 428 . In some embodiments, controller 460 includes programming (eg, instructions in a non-transitory computer readable medium) that adjusts the timing and provision of image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single monolithic device, or a distributed system connected by wired or wireless communication channels. In some embodiments, controller 460 may be part of processing module 260 or 270 (shown in FIG. 2 ).

The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each respective waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have other shapes (e.g., curved) having top and bottom major surfaces and edges extending between these top and bottom major surfaces. . In the configuration shown, the waveguides 440b, 438b, 436b, 434b, 432b may each include light extraction optics 440a, 438a, 436a, 434a, 432a configured to transmit The light from the is redirected out of the waveguide and the light is extracted out of the waveguide to output image information to the eye 410 . The extracted light may also be referred to as outcoupling light, and the light extraction optics may also be referred to as outcoupling optics. The extracted beam of light propagating in the waveguide illuminates the light redirecting element. The light extraction optics (440a, 438a, 436a, 434a, 432a) may, for example, be reflective or diffractive optical features. Although illustrated for ease of description and clarity of drawing as being disposed at the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b, in some embodiments the light extraction optics 440a, 438a, 436a, 434a , 432a may be disposed at the top and/or bottom major surfaces and/or may be disposed directly in the volume of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material that is attached to a transparent substrate to form waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b can be a single piece of material and the light extraction optics 440a, 438a, 436a, 434a, 432a can be formed on the surface of the piece of material or on the surface of the piece of material. in the interior.

Continuing with reference to FIG. 4, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light to form an image corresponding to a particular depth plane, as discussed herein. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light as injected into such waveguide 432b to the eye 410 . Collimated light can represent the optically infinite focal plane. The next upstream waveguide 434b may be configured to send the collimated light through the first lens 452 (eg, negative lens) out before it can reach the eye 410 . The first lens 452 may be configured to create a slightly convex wavefront curvature such that the eye/brain interprets the light from the next upstream waveguide 434b as coming from the first focal plane that is closer from optical infinity toward Inward toward the eye 410 . Similarly, third upstream waveguide 436b causes its output light to pass through first lens 452 and second lens 454 before reaching eye 410; the combined optical power of first and second lenses 452 and 454 can be configured To create another incremental wavefront curvature so that the eye/brain interprets light from the third waveguide 436b as coming from a second focal plane that is farther from optical infinity than light from the next upstream waveguide 434b Closer inward towards the person.

The other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the tallest waveguide 440b in the stack sending its output through all the lenses between it and the eye, used to represent the highest The aggregate focal power near the focal plane of the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing/interpreting light from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be provided at the top of the stack to compensate for the underlying lens stack 458, 456, 454, 452 convergent focus. This configuration provides as many perceived focal planes as there are waveguide/lens pairs available. The focusing aspects of the waveguide's light extraction optics and lenses may be static (eg, not dynamic or electroactive). In some alternative embodiments, either or both may be dynamic using electroactive features.

With continued reference to FIG. 4 , the light extraction optics 440a, 438a, 436a, 434a, 432a can be configured to redirect light out of their respective waveguides with an appropriate amount of divergence for the particular depth plane output associated with that waveguide. or a collimated amount of this light. As a result, waveguides with different associated depth planes may have different configurations of light extraction optics that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume or surface features that can be configured to output light at specific angles. For example, light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in US Patent Publication No. 2015/0178939, published June 25, 2015, which is incorporated herein by reference in its entirety.

In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features that form a diffractive pattern, or "diffractive optical elements" (also referred to herein as "DOEs"). Preferably, the DOE has a relatively low diffraction efficiency such that only a portion of the light of the beam is deflected towards the eye 410 by each intersection of the DOE, while the remainder continues to move through the waveguide via total internal reflection. The light carrying the image information can thus be split into a number of correlated exit beams that exit the waveguide at multiple locations, and the result is a fairly uniform pattern of exit towards the eye 304 for that particular collimated beam bouncing within the waveguide. emission.

In some embodiments, one or more DOEs may be switchable between their actively diffracting "on" state and their insignificantly diffracting "off" state. For example, a switchable DOE can include a polymer-dispersed liquid crystal layer in which the droplets contain a diffractive pattern in the host medium, and the refractive index of the droplets can be switched to substantially match that of the host material (in this case, pattern does not significantly diffract incident light), or the droplet can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

In some embodiments, the number and distribution of depth planes or depths of field can be changed dynamically based on the pupil size or orientation of the viewer's eyes. Depth of field can vary inversely proportional to the viewer's pupil size. As a result, as the size of the pupil of the viewer's eye decreases, the depth of field increases such that a plane cannot be discerned because the plane is located beyond the focal depth of the eye, and as the pupil size decreases and the corresponding depth of field increases, the plane may become Recognizable and appear more in focus. Likewise, the number of spaced apart depth planes used to present different images to the viewer may decrease as the pupil size decreases. For example, a viewer may not be able to clearly perceive details in a first depth plane and a second depth plane with one pupil size without accommodation of the eyes out of one depth plane and into another. However, these two depth planes can be fully in focus on the user at the same time with another pupil size without changing the accommodation.

In some embodiments, the display system may vary the number of waveguides receiving image information based on a determination of pupil size or orientation or upon receipt of an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eyes cannot distinguish between the two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) may be configured or programmed to stop the flow of water into the waveguides. One provides image information. Advantageously, this can reduce the processing load on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for the waveguide can be switched between on and off states, the DOE can be switched to the off state when the waveguide is actually receiving image information.

In some embodiments, it may be desirable for the exit beam to have a diameter smaller than the diameter of the viewer's eye. However, satisfying this condition can be challenging given the variation in pupil size of the viewer. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the size of the pupil decreases, the size of the exit beam may also decrease. In some embodiments, a variable aperture can be used to vary the exit beam size.

Wearable system 400 may include outward-facing imaging system 464 (eg, a digital camera) that images a portion of world 470 . This portion of world 470 may be referred to as the world camera's field of view (FOV), and imaging system 464 is sometimes referred to as a FOV camera. The FOV of the world camera may or may not be the same as the FOV of the viewer 210, which contains the portion of the world 470 that the viewer 210 perceives at a given time. For example, in some cases, the FOV of the world camera may be larger than the viewer 210 of the viewer 210 of the wearable system 400 . The entire area available for viewing or imaging by a viewer may be referred to as the field of regard (FOR). FOR can include a 4π steradian solid angle around the wearable system 400, since the wearer can move his body, head or eyes to perceive essentially any direction in space. In other contexts, the wearer's motion may be more narrow, so the wearer's FOR may subtend a smaller solid angle. As described with reference to FIG. 1B , user 210 may also have a FOV associated with the user's eyes when the user is using the HMD. In some embodiments, the FOV associated with the user's eyes may be the same as the FOV of imaging system 464 . In other embodiments, the FOV associated with the user's eyes is different than the FOV of the imaging system 464 . Images obtained from outward-facing imaging system 464 may be used to track gestures made by the user (eg, hand or finger gestures), detect objects in world 470 in front of the user, and the like.

Wearable system 400 may include an audio sensor 232, such as a microphone, to capture ambient sound. As noted above, in some embodiments, one or more other audio sensors may be positioned to provide stereo reception useful for determining the location of speech sources. As another example, audio sensor 232 may include a directional microphone, which may also provide useful directional information about the location of the audio source.

Wearable system 400 may also include an inward-facing imaging system 462 (eg, a digital camera) that observes the user's movements, such as eye movements and facial movements. Inwardly facing imaging system 462 may be used to capture images of eye 410 to determine the size and/or orientation of the pupil of eye 410 . Inwardly facing imaging system 462 may be used to obtain images for determining the direction the user is looking (eg, eye pose) or for biometric identification of the user (eg, via iris recognition). In some embodiments, each eye may utilize at least one camera to determine pupil size and eye pose for each eye individually, allowing image information to be presented to each eye to dynamically design for that eye. In some other embodiments, only the pupil diameter or orientation of a single eye 410 is determined (eg, using only a single camera per pair of eyes) and the user's eyes are assumed to be similar. Images obtained by inwardly facing imaging system 462 may be analyzed to determine the user's eye pose or emotion, which may be used by wearable system 400 to decide which audio or visual content should be presented to the user. Wearable system 400 may also use sensors such as IMUs, accelerometers, gyroscopes, etc. to determine head pose (eg, head position or head orientation).

Wearable system 400 may include a user input device 466 through which a user may input commands to controller 460 to interact with system 400 . For example, user input devices 466 may include trackpads, touch screens, joysticks, multiple degrees of freedom (DOF) controllers, capacitive sensing devices, game controllers, keyboards, mice, directional pads (D-pads), sticks ( wand), haptic devices, totems (eg, used as virtual user input devices), etc. A multi-DOF controller may sense some or all possible translations (e.g., left/right, forward/backward, or up/down) or rotation (e.g., yaw, pitch, or roll) of the controller (e.g., roll)) in the user input. A multi-DOF controller that supports translational motion may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (eg, thumb) to press or slide on a touch-sensitive input device to provide input to wearable system 400 (eg, to provide user input to a user interface provided by wearable system 400 ). During use of wearable system 400, user input device 466 may be held by a user's hand. User input device 466 may be in wired or wireless communication with wearable system 400 .

Fig. 5 shows an example of an outgoing beam output by a waveguide. One waveguide is shown, but it should be understood that the other waveguides in waveguide assembly 480, where waveguide assembly 480 includes a plurality of waveguides, may function similarly. Light 520 is injected into waveguide 432b at input edge 432c of waveguide 432b and propagates within waveguide 432b by TIR. At the point where light 5200 impinges on DOE 432a, a portion of the light, such as exit beam 510, exits the waveguide. Exit beam 510 is shown as being substantially parallel, but depending on the depth plane associated with waveguide 432b, exit beam 510 may also be redirected at an angle (eg, forming a diverging exit beam) for propagation to eye 410. It should be understood that a substantially parallel exit beam may be indicative of a waveguide having light extraction optics that couple light out to form image on the depth plane. Other waveguides or other sets of light extraction optics could output a more divergent exit beam pattern, which would require the eye 410 to accommodate closer to focus it on the retina and would be interpreted by the brain as coming from closer to the eye 410 than optical infinity distance light.

6 is a schematic diagram illustrating an optical system including a waveguide, an optical coupler subsystem to optically couple light to or from the waveguide, and a control subsystem for producing a multifocal volumetric display , image or light field. The optical system may include a waveguide, an optical coupler subsystem to optically couple light to or from the waveguide, and a control subsystem. The optical system can be used to generate multifocal volumes, images or light fields. The optical system may include one or more main planar waveguides 632a (only one is shown in FIG. 6 ) and one or more DOEs 632b associated with each of at least some of the main waveguides 632a. The planar waveguide 632b may be similar to the waveguides 432b, 434b, 436b, 438b, 440b discussed with reference to FIG. 4 . The optical system may employ a distributed waveguide to relay light along a first axis (vertical or Y axis in the view of FIG. 6) and expand the effective exit pupil of light along a first axis (eg, Y axis). The distributed waveguide arrangement may, for example, include a distributed planar waveguide 622b and at least one DOE 622a (shown by a two-dashed line) associated with the distributed planar waveguide 622b. The distributed planar waveguide 3 may be similar or identical to the main planar waveguide 1 in at least some respects, with a different orientation therefrom. Likewise, the at least one DOE 622a may be similar or identical to DOE 632a in at least some respects. For example, distribution planar waveguide 622b or DOE 622a may be constructed of the same material as main planar waveguide 632b or DOE 632a, respectively. The embodiment of the optical display system 600 shown in FIG. 6 may be integrated into the wearable system 200 shown in FIG. 2 .

Relay and exit pupil-expanded light may be optically coupled from the distributed waveguide arrangement into one or more principal planar waveguides 632b. The primary planar waveguide 632b can relay light along a second axis, which is preferably orthogonal to the first axis, (eg, consider the horizontal or X axis of FIG. 6). Notably, the second axis may be a non-orthogonal axis to the first axis. The principal planar waveguide 632b expands the effective exit pupil of light along a second axis (eg, the X-axis). For example, distribution planar waveguide 622b can relay and spread light along a vertical or Y axis and pass the light to main planar waveguide 632b which can relay and spread light along a horizontal or X axis.

The optical system can include one or more colored light sources (eg, red, green, and blue lasers) 610 that can be optically coupled to the proximal end of a single-mode fiber 640 . The distal end of the optical fiber 640 may be passed through or received by a hollow tube 642 of piezoelectric material. The distal end protrudes from the tube 642 as a fixed free flexible cantilever 644 . Piezoelectric tube 642 may be associated with four quadrant electrodes (not shown). For example, electrodes may be plated on the exterior, outer surface or perimeter or diameter of tube 642 . A core electrode (not shown) may also be located in the core, center, inner circumference or inner diameter of the tube 642 .

Drive electronics 650 , electrically coupled, eg, via wires 660 , drive opposing pairs of electrodes to bend piezoelectric tube 642 independently in two axes. The protruding distal tip of the optical fiber 640 has a mechanical resonance mode. The resonant frequency may depend on the diameter, length and material properties of the optical fiber 640 . By vibrating the piezoelectric tube 642 near the first mechanical resonance mode of the fiber cantilever 644, the fiber cantilever 644 can be vibrated and large deflection can be swept.

The tip of the fiber optic cantilever 644 is scanned biaxially in an area-filling two-dimensional (2D) scan by stimulating resonances in both axes. By modulating the intensity of the light source 610 synchronously with the scanning of the fiber optic cantilever 644, the light emerging from the fiber optic cantilever 644 can form an image. A description of such an arrangement is provided in US Patent Publication No. 2014/0003762, which is hereby incorporated by reference in its entirety.

Components of the optical coupler subsystem may collimate the light exiting the scanning fiber cantilever 644 . The collimated light may be reflected by a mirror 648 into a narrow distribution planar waveguide 622b comprising at least one diffractive optical element (DOE) 622a. Collimated light may propagate vertically (with respect to the view of FIG. 6 ) along the distribution planar waveguide 622b by TIR, thereby repeatedly intersecting the DOE 622a. DOE 622a preferably has low diffraction efficiency. This causes a portion (e.g., 10%) of the light to be diffracted at each intersection with the DOE 622a towards the edge of the larger principal planar waveguide 632b, and the portion of the light to continue on its original trajectory down the distribution plane via TIR The length of the waveguide 622b.

At each intersection with DOE 622a, additional light may be diffracted towards the entrance of main waveguide 632b. By splitting the incident light into multiple outcouple groups, the exit pupil of the light can be expanded vertically by the DOE 622a in the distribution planar waveguide 622b. This vertically expanding light coupled out of the distribution planar waveguide 622b can enter the edge of the main planar waveguide 632b.

Light entering the main waveguide 632b may propagate horizontally along the main waveguide 632b via TIR (with respect to the view of FIG. 6 ). As the light intersects DOE 632a at multiple points, it propagates horizontally via TIR along at least part of the length of main waveguide 632b. The DOE 632a may advantageously be designed or configured to have a phase profile that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern to produce deflection and focusing of light. The DOE 632a may advantageously have a low diffraction efficiency (eg, 10%) such that at each intersection of the DOE 632a only a portion of the light beam is deflected towards the viewing eye, while the rest of the light continues to propagate through the main waveguide 632b via TIR.

At each intersection between the propagating light and the DOE 632a, a portion of the light diffracts toward and exits the face of the main waveguide 632b which allows the light to escape from the TIR. In some embodiments, the radially symmetric diffraction pattern of DOE 632a additionally imparts a focus level of the diffracted light, shapes (eg, imparts curvature) the optical wavefronts of individual beams, and steers the beams at angles that match the designed focus level.

Thus, these different paths can cause light to be coupled out of the main planar waveguide 632b through multiple DOEs 632a that have different angles, focus levels, or produce different fill patterns at the exit pupil. Different fill patterns at the exit pupil can be advantageously used to create light field displays with multiple depth planes. Each layer in the waveguide assembly or a set of layers (eg 3 layers) in the stack can be used to generate a corresponding color (eg red, blue, green). Thus, for example, a first set of three adjacent layers may be employed to generate red, blue and green light, respectively, at a first depth of focus. A second set of three adjacent layers may be employed to generate red, blue and green light, respectively, at a second depth of focus. Multiple groups can be employed to generate a full 3D or 4D color image light field with various depths of focus.

Other components of the wearable system

In many implementations, a wearable system may include other components in addition to or instead of the components of the wearable system described above. A wearable system may, for example, include one or more haptic devices or components. Haptic devices or components can be used to provide a sense of touch to a user. For example, a haptic device or component may provide a haptic sensation of pressure or texture when virtual content (eg, a virtual object, virtual tool, other virtual construct) is touched. Haptics may replicate the feel of a physical object represented by the virtual object or may replicate the feel of an imaginary object or character (eg, a dragon) represented by the virtual content. In some implementations, a user may wear a haptic device or assembly (eg, a user-wearable glove). In some implementations, a haptic device or assembly can be held by a user.

A wearable system may, for example, include one or more physical objects that may be manipulated by a user to allow input or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects such as, for example, the surface of a piece of metal or plastic, a wall, or a table. In some implementations, the totem may not actually have any physical input structures (eg, keys, triggers, joysticks, trackballs, rocker switches). Instead, the totem can provide only physical surfaces, and the wearable system can present a user interface so that the user appears to be on one or more surfaces of the totem. For example, the wearable system can render images of a computer keyboard and trackpad to appear to be on one or more surfaces of the totem. For example, a wearable system could make a virtual computer keyboard and a virtual trackpad appear to be on the surface of a rectangular thin aluminum plate used as a totem. The rectangular pad itself doesn't have any physical keys or trackpad or sensors. However, due to selection or input via the virtual keyboard and/or virtual touchpad, the wearable system can detect user manipulation or interaction or touch by means of the rectangular pad. User input device 466 (shown in FIG. 4 ) may be an embodiment of a totem, which may include a touchpad, touch pad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller or other physical input device. Users can use totems alone or in combination with gestures to interact with the wearable system or other users.

Examples of haptic devices and totems that may be used with the wearable devices, HMDs, and display systems of the present disclosure are described in US Patent Publication No. 2015/0016777, which is incorporated herein by reference in its entirety.

Example wearable systems, environments and interfaces

Wearable systems can employ various mapping-related techniques in order to achieve a high depth of field in the presented light field. When mapping out the virtual world, it is advantageous to know all the features and points in the real world to accurately depict virtual objects relative to the real world. To this end, FOV images captured from the user of the wearable system can be added to the world model by including new pictures that convey information about various points and features of the real world. For example, a wearable system can collect sets of mapping points (such as 2D points or 3D points) and find new mapping points to present a more accurate version of the world model. A first user's world model can be communicated (eg, via a network such as a cloud network) to a second user so that the second user can experience the world around the first user.

FIG. 7 is a block diagram of an example of an MR environment 700 . MR environment 700 may be configured to receive input from one or more user wearable systems (e.g., wearable system 200 or display system 220) or fixed indoor systems (e.g., indoor cameras, etc.) visual input to the system 702, fixed input such as an indoor camera 704, sensory input 706 from various sensors, gestures, totems, eye tracking, user input from user input device 466, etc.). Wearable systems can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, motion sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and location of the user's environment. various other properties. This information can be further supplemented with information from fixed cameras in the room, which can provide images or various cues from different viewpoints. Image data acquired by a camera (such as an indoor camera and/or a camera of an outward-facing imaging system) may be reduced to a set of map points.

One or more object recognizers 708 may crawl through the received data (eg, collections of points) and with the help of mapping database 710 recognize or map points, label images, attach semantic information to objects. Mapping database 710 may include various points and their corresponding objects collected over time. Various devices and mapping databases can be connected to each other through a network (eg, LAN, WAN, etc.) to access the cloud.

Based on this information and the set of points in the mapping database, object identifiers 708a through 708n can identify objects in the environment. For example, an object recognizer may recognize faces, people, windows, walls, user input devices, televisions, documents (eg, travel tickets, driver's licenses, passports as described in the security examples herein), other objects in the user's environment, and the like. One or more object discriminators can be dedicated to objects with certain properties. For example, object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize documents.

Object recognition can be performed using various computer vision techniques. For example, the wearable system can analyze images acquired by outward-facing imaging system 464 (shown in FIG. 4 ) to perform scene reconstruction, event detection, video tracking, object recognition (e.g., people or documents), object pose estimation, facial Recognition (e.g., from people in the environment or images on documents), learning, indexing, motion estimation, or image analysis (e.g., recognizing indicia within documents, such as photographs, signatures, identification information, travel information, etc.) etc. These tasks can be performed using one or more computer vision algorithms. Non-limiting examples of computer vision algorithms include: scale (scale) invariant feature transform (SIFT), accelerated robust (robust) feature (SURF), orientation (orient) FAST and rotation BRIEF (ORB), binary robust invariant variable Scaled Keypoint (BRISK), Fast Retinal Keypoint (FREAK), Viola-Jones Algorithm, Eigenfaces (Eigenfaces) Method, Lucas-Kanade Algorithm, Horn - Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) technique, Sequential Bayesian estimators (e.g., Kalman filter, Extended Kalman filters, etc.), bundle adjustment, self-adjusting thresholding (and other thresholding techniques), iterative closest point (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histogram, various machine learning algorithms (such as support vector machines, k-nearest neighbors, naive Bayesian, neural networks (including convolutional or deep neural networks), or other supervised/unsupervised models, etc.), etc.

One or more object recognizers 708 may also implement various text recognition algorithms to recognize and extract text from images. Some example text recognition algorithms include: optical character recognition (OCR) algorithms, deep learning algorithms (eg, deep neural networks), pattern matching algorithms, algorithms for preprocessing, and the like.

Additionally or alternatively, object recognition can be performed by various machine learning algorithms. Once trained, the machine learning algorithm can be stored by the HMD. Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression), instance-based algorithms (e.g., learning vector quantization), decision tree algorithms (e.g., classification and regression trees), Bayesian algorithms (e.g., Naive Bayes), clustering algorithms (e.g., k-means clustering), association rule learning algorithms (e.g., a-priori algorithms), artificial neural network algorithms (e.g., perceptrons), deep learning algorithms (e.g., deep Boltzmann machines or deep neural networks), dimensionality reduction algorithms (e.g., principal component analysis), ensemble algorithms (e.g., stacked generalization), or other machine learning algorithm. In some embodiments, individual models can be customized for individual data sets. For example, a wearable device may generate or store base models. The base model can be used as a starting point to generate data-type-specific (e.g., a particular user in a telepresence session), data group (e.g., a set of acquired additional images of a user in a telepresence session), conditional situation, or other variants additional models. In some embodiments, a wearable HMD can be configured to utilize a variety of techniques to generate models for analyzing aggregated data. Other techniques may include using pre-defined thresholds or data values.

Based on this information and the set of points in the mapping database, object identifiers 708a through 708n can identify objects and supplement them with semantic information to animate them. For example, if an object discriminator recognizes a group of points as a door, the system can attach some semantic information (eg, a door has a hinge and has a 90 degree motion around the hinge). If the object discriminator recognizes the group of points as a mirror, the system can attach semantic information that the mirror has a reflective surface that can reflect images of objects in the room. Semantic information may include functional visibility of objects as described herein. For example, semantic information may include normals of objects. The system can assign a vector whose direction indicates the normal of the object. In some implementations, once the object recognizer 708 has recognized an environment (e.g., a leisure or work environment, a public or private environment, or a home environment, etc.) Associated with some coordinates in world map or GPS coordinates. For example, once the wearable system recognizes (e.g., through object recognizer 708 or the user's response) that the environment is a living room in the user's home, the wearable system can automatically correlate the location of the environment with GPS coordinates or with a location on a world map couplet. As a result, the wearable system can present/block virtual content based on the living room environment when the user enters the same location in the future. The wearable system can also create settings for silencing the wearable device or for presenting customized content for the discerned environment as part of the environment's semantic information. Therefore, when the user enters the same location in the future, the wearable system can automatically present virtual content or silence the wearable device according to the environment without re-discriminating the type of environment, which can improve efficiency and reduce latency.

Over time, the mapping database grows as the system (which can reside locally or be accessed over a wireless network) accumulates more data from the world. Once an object is identified, the information can be sent to one or more wearable systems. For example, MR environment 700 may include information about a scene taking place in California. Environment 700 may be sent to one or more users in New York. Based on data received from FOV cameras and other inputs, object discriminators and other software components can map points collected from various images, identify objects, etc., so that the scene can be accurately "delivered" to the second party, which may be located in a different part of the world. Second user. Environment 700 may also use topology maps for localization purposes.

8 is a process flow diagram of an example of a method 800 of presenting virtual content related to a recognized object. Method 800 describes how a virtual scene is presented to a user of a wearable system. Users may be geographically distant from the scene. For example, a user may be located in New York, but may want to see a scene currently taking place in California or may want to go for a walk with a friend who lives in California.

At block 810, the wearable system may receive input from the user and other users regarding the user's environment. This can be achieved through various input means and mapping the knowledge already held in the database. At block 810, the user's FOV camera, sensors, GPS, eye tracking, etc. communicate information to the system. At block 820, the system may determine sparse points based on this information. The sparse points can be used to determine pose data (eg, head pose, eye pose, body pose, or hand pose) that can be used to display and understand the orientation and position of various objects in the user's surrounding environment. At block 830, the object recognizers 708a-708n may crawl through the collected points and recognize one or more objects using the mapping database. Then, at block 840, this information can be communicated to the user's personal wearable system, and at block 850, the desired virtual scene can be displayed to the user accordingly. For example, a desired virtual scene (eg, a user in CA) may be displayed in an appropriate orientation, position, etc. relative to various objects and the user's other surroundings in New York.

9 is a block diagram of another example of a wearable system. In this example, wearable system 900 includes mapping 920, which may include mapping database 710 containing mapping data about the world. The map may reside partly locally on the wearable system and partly in a network storage location (eg, in a cloud system) accessible by a wired or wireless network. Gesture process 910 may execute on the wearable computing architecture (eg, processing module 260 or controller 460 ) and utilize data from 920 maps to determine the position and orientation of the wearable computing hardware or the user. Pose data can be computed from data collected on the fly as the user is experiencing the system and operating in the world. Data may include images, data from sensors such as inertial measurement units, which typically include accelerometer and gyroscope components, and surface information related to objects in real or virtual environments.

Sparse point representations can be the output of simultaneous localization and mapping (eg, SLAM or vSLAM, which refers to configurations where the input is only images/visions) processes. The system can be configured to not only find out where various components of the world are located, but also what the world is made of. Gestures can be building blocks for many purposes, including populating maps and using data from maps.

In one embodiment, sparse point locations by themselves may not be entirely sufficient and further information may be required to produce a multi-focal AR, VR or MR experience. A dense representation, usually represented as depth map information, can be used to at least partially fill this gap. Such information may be calculated from a process called stereoscopic process 940, where depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and motion patterns (such as infrared patterns created using motion projectors), images captured from image cameras, or hand gestures/totems 950 may be used as input to the stereoscopic process 940 . A large amount of depth map information can be fused together, and some of it can be generalized with a surface representation. For example, mathematically definable surfaces can be efficient (eg, with respect to large point clouds) and can be ingestible input to other processing devices, such as game engines. Thus, the output of the stereo process 940 (eg, the depth map) may be combined in the fusion process 930 . Pose 910 may also be an input to this fusion process 930 , and the output of fusion 930 becomes an input to fill-mapping process 920 . Subsurfaces can be connected to each other, such as in terrain maps, to form larger surfaces, and the map becomes a large mixture of points and surfaces.

To address various aspects in the mixed reality process 960, various inputs may be used. For example, in the embodiment shown in FIG. 9, a game parameter may be an input to determine that a user of the system is fighting one or more monsters at various locations, dying under various conditions, or fleeing (such as if the user shoots a monster). Play monster battle games against monsters, walls, or other objects etc. at different locations. A world map can include information about the location of objects or semantic information of objects, and is another valuable input to mixed reality. Pose relative to the world also becomes an input and plays a key role in almost any interactive system.

Control or input from a user is another input to wearable system 900 . As described herein, user input may include visual input, gestures, totems, audio input, sensory input, and the like. In order to move around or play a game, for example, the user may need to indicate to the wearable system 900 what he or she wants to do. In addition to moving oneself in space, various forms of user control can be used. In one embodiment, a totem (eg, a user input device) or an object such as a toy gun can be held by the user and tracked by the system. The system will preferably be configured to know that the user is holding an item and understand what kind of interaction the user is having with the item (e.g. if the totem or object is a gun, the system can be configured to understand position and orientation and whether the user is clicking the trigger or other sensing buttons or elements that may be equipped with sensors such as an IMU, which may help determine what is going on, even if such activity is not within the field of view of any cameras).

Hand gesture tracking or discrimination can also provide input. Wearable system 900 may be configured to track and interpret hand gestures for button presses, for gesturing left or right, stopping, grabbing, holding, and the like. For example, in one configuration, a user may want to flip through email or a calendar or do "fist bumps" with other people or players in a non-gaming environment. Wearable system 900 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, hand gestures can be simple static gestures such as open hand for stop, thumbs up for ok, thumbs down for bad, or hand gestures for directional commands Flip right or left or up/down.

Eye tracking is another input (eg, tracking where the user is looking to control display technology to render at a particular depth or range). In one embodiment, triangulation can be used to determine the vergence of the eyes, and then using a vergence/accommodation model developed for that particular person, accommodation can be determined. Eye tracking can be performed by an eye camera to determine eye gaze (eg, direction or orientation of one or both eyes). Other techniques can be used for eye tracking, such as, for example, measuring electrical potential by electrodes placed near the eye (eg, electro-oculography).

Voice tracking may be another input that may be used alone or in combination with other inputs (eg, totem tracking, eye tracking, gesture tracking, etc.). Voice tracking may include voice recognition, voice recognition, or a combination thereof. System 900 may include an audio sensor (eg, a microphone) that receives an audio stream from an environment. System 900 may incorporate speaker recognition techniques to determine who is speaking (e.g., whether the voice is from the wearer of the wearable device or from another person or voice (e.g., recorded voice sent by a speaker in the environment)), and Speech recognition technology can be incorporated to determine what was said. The local data and processing module or the remote processing module 270 may process audio data from the microphone (or audio data in another stream, e.g., a video stream that the user is watching) to recognize the content of the speech by applying various speech recognition algorithms, Various speech recognition algorithms such as hidden Markov models, dynamic time warping (DTW) based speech recognition, neural networks, deep learning algorithms such as deep feed-forward and recurrent neural networks, end-to-end automatic speech recognition, machine learning algorithms ( 7) or use other algorithms such as acoustic modeling or language modeling.

Another input to the mixed reality process 960 may include event tracking. Data acquired from outward-facing imaging system 464 can be used for event tracking, and the wearable system can analyze such imaging information (using computer vision techniques) to determine whether a triggering event is occurring, which can beneficially cause the system to automatically Silence the visual or audio content being presented to the user.

The local data and processing module or the remote processing module 270 may also apply voice recognition algorithms that can identify the speaker, eg, whether the speaker is the user 210 of the wearable system 900 or someone else the user is talking to. Some exemplary speech recognition algorithms may include frequency estimation, hidden Markov models, Gaussian mixture models, pattern matching algorithms, neural networks, matrix representations, vector quantization, speaker classification (diarisation), decision trees, and dynamic time warping (DTW) technology. Speech recognition techniques may also include anti-speaker techniques, such as group models and world models. Spectral features can be used to represent speaker characteristics. The local data and processing module or the remote data processing module 270 may use various machine learning algorithms described with reference to FIG. 7 to perform speech recognition.

Regarding the camera system, the exemplary wearable system 900 shown in FIG. 9 includes three pairs of cameras: a relatively wide FOV or passive SLAM camera pair arranged to the side of the user's face; a different camera oriented in front of the user. Pair to handle the stereoscopic imaging process and also for capturing hand poses and totem/object tracking in front of the user's face. The FOV camera or pair of cameras used for stereoscopic process 940 may also be referred to as camera 16 . The FOV camera and camera pair for stereoscopic process 940 may be part of outward facing imaging system 464 (shown in FIG. 4 ). Wearable system 900 may include an eye-tracking camera (also shown as eye camera 24 and which may be part of inward-facing imaging system 462 shown in FIG. Additional information is triangulated. Wearable system 900 may also include one or more texture light projectors, such as infrared (IR) projectors, to inject texture into the scene.

Example of a wearable system including environmental sensors

10 shows a schematic diagram of an example of various components of a wearable system including environmental sensors. In some embodiments, augmented reality display system 1010 may be an embodiment of the display system shown in FIG. 2 . In some implementations, AR display system 1010 may be a mixed reality display system. Environmental sensors may include sensors 24 , 28 , 30 , 32 and 34 . Environmental sensors may be configured to detect data about a user of the AR system (also referred to as user sensors) or to collect data about the user's environment (also referred to as external sensors). For example, a physiological sensor may be an embodiment of a user sensor, while a barometer may be an external sensor. In some cases, the sensors may be user sensors and external sensors. For example, when a user is positioned in front of a reflective surface (eg, a mirror), an outward-facing imaging system may acquire images of the user's environment as well as images of the user. As another example, a microphone can be used as both a user sensor and an external sensor, since the microphone can pick up sounds from the user and the environment. In the example shown in FIG. 10 , sensors 24 , 28 , 30 , and 32 may be user sensors, while sensor 34 may be an external sensor.

As shown, the augmented reality display system 1010 may include various user sensors. Augmented reality display system 1010 may include viewer imaging system 22 . Viewer imaging system 22 may be an embodiment of inward facing imaging system 462 depicted in FIG. 4 . The viewer imaging system 22 may include a camera 24 (e.g., an infrared, UV, other invisible and/or visible light camera) paired with a light source 26 (e.g., an infrared light source) directed at the user and configured to monitor the user (e.g., Eyes 1001, 1002 and/or surrounding tissue of the user). Camera 24 and light source 26 may be operatively coupled to a local processing module 270 . Such cameras 24 may be configured to monitor one or more of the orientation, shape, and symmetry of the pupil (including pupil size), iris, and/or tissues surrounding the eye (such as eyelids or eyebrows) of the respective eye for purposes of this disclosure. public analysis. In some embodiments, imaging of the iris and/or retina of the eye may be used for secure identification of the user. With continued reference to FIG. 10 , the cameras 24 may be further configured to image the retina of the respective eye, such as for diagnostic purposes and/or for orientation tracking based on the location of retinal features such as the fovea or features of the fundus. Iris and retinal imaging or scanning may be performed for secure identification of users, for example, to correctly associate user data with a particular user and/or present private information to appropriate users. In some embodiments, one or more cameras 28 may be configured in addition to or instead of cameras 24 to detect and/or monitor various other aspects of a user's state. For example, one or more cameras 28 may be inward-facing and configured to monitor the shape, position, motion, color, and/or other characteristics of features other than the user's eyes, for example, one or more facial features (e.g., , facial expressions, voluntary movements, involuntary tics). In another example, one or more cameras 28 may be downward facing or outward facing and configured to monitor the arms, hands, legs, feet of the user, another person in the user's FOV, an object in the FOV, etc. and/or the position, movement and/or other characteristics or properties of the torso. Camera 28 may be used to image the environment, and the wearable device may analyze such images to determine whether a triggering event is occurring such that visual or audible content being presented to the user by the wearable device should be silenced.

In some embodiments, as disclosed herein, display system 1010 may include a spatial light modulator that is variably A beam of light is projected across the user's retina to form an image. In some embodiments, a fiber optic scanner may be used in conjunction with or instead of cameras 24 or 28, for example, to track or image a user's eye. For example, instead of or in addition to a scanning optical fiber configured to output light, the health system may have a separate light receiving device to receive light reflected from the user's eye and collect data associated with the reflected light.

With continued reference to FIG. 10 , the cameras 24 , 28 and the light source 26 may be mounted on a frame 230 which may also hold the waveguide stacks 1005 , 1006 . In some embodiments, sensors and/or other electronics (eg, cameras 24 , 28 and light sources 26 ) of display system 1010 may be configured to communicate with local processing and data module 270 via communication links 262 , 264 .

In some embodiments, in addition to providing data about the user, one or both of cameras 24 and 28 may be utilized to track eyes to provide user input. For example, viewer imaging system 22 may be used to select items on a virtual menu and/or provide other input to display system 2010, such as to provide user responses in the various tests and analyzes disclosed herein.

In some embodiments, display system 1010 may include motion sensors 32, such as one or more accelerometers, gyroscopes, posture sensors, gait sensors, balance sensors, and/or IMU sensors. Sensor 30 may include one or more inwardly-pointing (user-pointing) microphones configured to detect sounds and various attributes of those sounds, including the intensity and type of detected sounds, the presence of multiple signals, and/or signal location.

Sensor 30 is shown schematically connected to frame 230 . It should be understood that this connection may take the form of a physical attachment to the frame 230, and may be anywhere on the frame 230, including the ends of the temples of the frame 230 that extend over the user's ear. For example, the sensor 30 may be mounted at the end of the leg of the frame 230 at a point of contact between the frame 230 and the user. In some other embodiments, sensor 30 may extend away from frame 230 to contact user 210 . In other embodiments, sensor 30 may not be physically attached to frame 230 ; however, sensor 30 may be spaced apart from frame 230 .

In some embodiments, display system 1010 may also include one or more environmental sensors 34 configured to detect objects, stimuli, people, animals, locations, or other aspects of the world around the user. For example, environmental sensors 34 may include one or more cameras, altimeters, barometers, chemical sensors, humidity sensors, temperature sensors, external microphones, light sensors (e.g., photometers), timing devices (e.g., clocks or calendars), or any combination or sub-combination. In some embodiments, multiple (eg, two) microphones may be spaced apart to facilitate sound source location determination. In various embodiments including an environment sensing camera, the camera may be positioned, eg, facing outward, so as to capture an image similar to at least a portion of the user's normal field of view. The environmental sensor may also include a transmitting device configured to receive signals such as laser light, visible light, invisible light wavelengths, sound (eg, audible sound, ultrasound, or other frequencies). In some embodiments, one or more environmental sensors (eg, cameras or light sensors) may be configured to measure ambient light (eg, brightness) of the environment (eg, to capture lighting conditions of the environment). Physical contact sensors such as strain gauges, curb detectors, etc. may also be included as environmental sensors.

In some embodiments, display system 1010 may also be configured to receive other environmental inputs, such as GPS location data, weather data, date and time, or data that may be received from the Internet, satellite communications, or other suitable wired or wireless data communication methods. Other available environment data. The processing module 260 may be configured to access additional information characterizing the user's location, such as pollen counts, demographics, air pollution, environmental toxicants, information from smart thermostats, lifestyle statistics, or other user, building, or health care providers near. In some embodiments, the information characterizing the location may be accessed using a cloud-based or other remote database. The processing module 260 may be configured to obtain such data and/or further analyze data from any one or combination of environmental sensors.

Display system 1010 may be configured to collect and store data obtained through any of the aforementioned sensors and/or inputs for an extended period of time. Data received at the device may be processed and/or stored locally at the processing module and/or remotely (eg, at the remote processing module 270 or at the remote data repository 280 as shown in FIG. 2 ). In some embodiments, additional data may be received directly at the local processing module, such as date and time, GPS location, or other global data. Data regarding content such as images, other visual content or audio content delivered to the user by the system may also be received at the local processing module.

Automatic Control of Wearable Display Systems

As noted above, situations may arise where it is desirable or even necessary to de-emphasize or block virtual content, or even turn off the wearable device's display of virtual content. This can happen in response to a triggering event, such as an emergency situation, an unsafe situation, or perhaps a need to present less virtual content to the user of the wearable device so that the user can focus more on the physical world outside of the user. Condition. Triggering events can also be based on the environment in which the user is using the system. Wearable systems can block virtual content or present customized virtual content based on the user's environment. For example, a wearable system could block a video game if the wearable system detects that the user is working.

Embodiments of wearable devices disclosed herein may include components and functionality capable of determining whether this is occurring and taking appropriate action to silence the wearable system, such as by silencing virtual content (e.g., de-emphasizing, blocking, or turning off virtual content). content) or by silencing one or more components of the wearable system (eg, turning off one or more components, dimming one or more components, putting one or more components into sleep mode). As used herein, silencing virtual content may generally include de-emphasizing, attenuating, or reducing the amount or impact of visual or auditory content presented to a user by a wearable device, up to and including turning off the content. Silencing may include visible silencing (eg, turning off or dimming display 220 ) or audible silencing (eg, reducing sound from speaker 240 or turning off the speaker completely). Silencing may include increasing the transparency of visible virtual content, which makes it easier for a user to see through such virtual content to perceive the external physical world. Silencing may also include reducing the size of the virtual content or changing its placement so that it is less prominent in the user's field of view. Silencing may also include blocking content from the wearable device's display or selectively allowing some content but not others. Thus, silencing can be achieved through a blacklist (which identifies content to be blocked) or through a whitelist (which identifies content to allow). In some implementations, a combination of blacklists and whitelists can be used to effectively silence content. Additionally or alternatively, greylisting can be used to indicate content that should be temporarily blocked (or allowed) until another condition or event occurs. For example, in an office environment, certain virtual content can be greylisted and temporarily blocked from being displayed to the user until the user's administrator overrides the blocking and moves the content to the whitelist or permanently by moving the content to the blacklist to block content. Various embodiments of wearable devices described herein may use some or all of the aforementioned techniques to silence virtual content presented to a user.

In the following, various non-limiting illustrative examples of user experiences will be described in which it may be desirable to silence virtual content. Following these examples, techniques and devices for determining that an event is occurring that triggers a wearable device to silence virtual content will be described.

Example of Silencing a Wearable Device in a Surgical Context

11A and 11B illustrate an example of silencing an HMD in a surgical setting. In FIG. 11A , a surgeon is operating on a heart 1147 . A surgeon may wear an HMD as described herein. The surgeon can feel the heart 1147 in his FOV. The surgeon can also perceive virtual objects 1141, 1142 and 1145 in his FOV. Virtual objects 1141, 1142, and 1145 may be related to various metrics (eg, heart rate, ECG, etc.) and diagnoses (eg, arrhythmia, cardiac arrest, etc.) associated with the heart. The HMD may present virtual objects 1141 , 1142 , and 1145 based on information acquired by environmental sensors of the wearable system or by communicating with another device or remote processing module of the wearable system.

During surgery, however, accidents or emergencies may occur. For example, there may be a sudden, undesired flow of blood at the surgical site (as shown by the jet of blood 1149 from the heart 1147 in FIG. 11B ). A wearable system can detect this using computer vision techniques, for example, by detecting (in images acquired by an outward-facing camera) rapidly occurring changes in key points or features in or near the surgical site. The wearable system can also make detections based on data received from other devices or remote processing modules.

The wearable system can determine that the condition meets the criteria for a triggering event, where the display of visual or auditory virtual content should be silenced so that the surgeon can focus on the unexpected or emergency situation. Accordingly, the wearable system may automatically silence the virtual content in response to automatic detection of a triggering event (in this example, the spurt of blood 1149). As a result, in FIG. 11B , the HMD does not present virtual objects 1141 , 1142 , and 1145 to the surgeon, and the surgeon can focus all his attention on stopping the eruption of blood.

The HMD may resume normal operation and resume presenting virtual content to the surgeon in response to the termination event. Termination events may be detected when the triggering event ends (eg, blood stops spraying) or when the user enters another environment in which no triggering event exists (eg, when the user walks out of the emergency room). Termination events can also be based on a threshold time period. For example, the HMD may resume normal operation after detecting the triggering event for a period of time (eg, 5 minutes, 15 minutes, 1 hour, etc.) or after detecting the end of the triggering event for a period of time. In this example, the wearable system may restore the display (or other components of the wearable system) before the trigger event ends.

Example of silencing a wearable device in an industrial context

Similar techniques for silencing HMDs can also be applied in other contexts. For example, these techniques can be used in industrial settings. As an example, a worker may weld a metal workpiece in a factory while wearing an HMD. Through the HMD, the worker can perceive the metal he is working on and the virtual content associated with the welding process. For example, an HMD can display virtual content, including instructions on how to solder parts.

However, accidents or emergencies may occur while workers are using the HMD. For example, a worker's clothing could accidentally catch fire or a welding torch could overheat or set fire to the workpiece or nearby materials. Other emergencies may occur, such as industrial chemical spills in the worker environment. The wearable system can detect these situations as events that trigger the HMD to silence the virtual content. As further described with reference to FIGS. 12A-12C , the wearable system can use computer vision algorithms (or machine learning algorithms) to detect trigger events by analyzing images of the worker's environment. For example, to detect fire or overheating, a wearable system could analyze infrared (IR) images taken by an outward-facing camera, since heat from a fire or overheating will be particularly evident in the IR image. The wearable system may automatically silence the display of virtual content in response to detecting a trigger event. In some cases, the wearable system may provide a warning indicating that the HMD will automatically shut down unless otherwise instructed by the user.

In some embodiments, a worker can manually actuate the reality button 263, which can cause the HMD to silence the virtual content. For example, a worker could sense an emergency or unsafe condition (eg, by smelling overheated material) and actuate a reality button so that the worker can more easily focus on the actual reality. To avoid accidentally silencing virtual content when the worker is still interested in the virtual content, the HMD may provide a warning to the worker before performing the silence operation. For example, upon detecting the actuation of the reality button, the HMD may provide the worker with a message indicating that the virtual content will be silenced soon (e.g., within seconds), unless the worker instructs otherwise (e.g., by actuating the reality button again). button or change his pose). Further details regarding such warnings are described below with reference to Figures 14A and 14B.

Figure 11C shows a gardener operating a machine such as a lawn mower. Like many repetitive jobs, mowing can be tedious. After a period of time, workers may lose interest, increasing the likelihood of accidents. Furthermore, it may be difficult to attract qualified staff or ensure adequate staff performance.

The worker shown in FIG. 11C wears an HMD that presents virtual content 1130 in the user's field of view to enhance work performance. For example, as shown in scene 1100c, the HMD may present a virtual game where the object is to follow a virtual map pattern. Points are received for accurately following patterns and hitting certain score bonuses before they disappear. Points are deducted for deviating from the pattern or deviating too close to certain physical objects (eg, trees, sprinklers, roads).

However, a worker may encounter an incoming vehicle, which may be traveling at a very fast speed, or a pedestrian, who may be walking in front of the machine. Staff may need to react to the entering vehicle or pedestrian (eg by slowing down or changing direction). The wearable system can use its outward-facing imaging system to acquire images of the worker's surroundings and use computer vision algorithms to detect entering vehicles or pedestrians.

The wearable system can calculate speed or distance from the worker based on the captured images (or location-based data from other environmental sensors such as GPS). If the wearable system determines that the speed or distance exceeds threshold conditions (e.g., a vehicle is approaching very quickly or a vehicle or pedestrian is very close to a worker), the HMD can automatically silence the virtual content (e.g., by pausing the game, moving the virtual game to FOV outside) to reduce distraction and allow workers to concentrate on maneuvering the mower to avoid entering vehicles or pedestrians. For example, as shown in scene 1132c, when the HMD silences the virtual content, the user does not perceive virtual game piece 1130.

When the wearable system detects a terminating condition, for example, when the triggering event ends, the HMD can resume normal operation and resume presenting virtual content to workers. In some implementations, the HMD can silence the virtual content while the rest of the HMD can continue to operate. For example, a wearable system may continuously image the user's location using one or more environmental sensors, such as GPS or an outward-facing camera. When the wearable system determines that a vehicle has entered or that a pedestrian has passed a worker, the wearable system can reopen the virtual content.

In some implementations, the wearable system can present a warning before resuming normal operation or resuming presentation of virtual content. This may prevent virtual content from opening while the triggering event is still in progress if the user needs time to recover after an emergency or for any other reason (eg, while the user is still in an emergency). In response to the warning, the user may actuate the reality button 263 if the user wishes the virtual content to remain silent. In some implementations, a user may resume virtual content, either through manual user input or automatically, during a triggering event. This allows for situations where virtual content can assist the user during a triggering event. For example, the system could automatically detect if a child is choking, thereby silencing a parent's virtual content. Where the system has an emergency response application installed, the system may automatically selectively enable only virtual content related to the emergency response application if the parent does not respond within a threshold period of time or if the parent does not take corrective action.

Examples of Silencing Wearables in an Educational Context

Figure 1 ID shows an example of silencing an HMD in an educational context. FIG. 11D shows a classroom 1100d in which two students 1122 and 1124 are physically seated in the classroom (in this example, the class is a yoga class). When students 1122 and 1124 wear the HMD, they can perceive a virtual avatar for the student 1126 and a virtual avatar for the teacher 1110, neither of whom are physically present in the room. A student can attend classes at his home (rather than in classroom 1100d).

In one instance, a student 1122 may want to discuss course-related issues (eg, how to perform a particular yoga pose) with other students 1124 during the class. Student 1122 can walk to Student 1124. The wearable system of student 1124 can detect that another student 1122 is in front of her and automatically silence the audio and virtual content presented by the HMD to allow students 1124 and 1122 to interact in person with less (or no) virtual content presented. For example, a wearable system may use a facial recognition algorithm to detect the presence of a natural person in front of the HMD (which may be an example of a trigger event for the HMD to automatically silence virtual content). In response to this detection, the HMD may turn off (or attenuate) audio and virtual content from the HMD. In the example shown in FIG. 11D , once student 1124's HMD is silenced, student 1124 will not be able to perceive avatars 1126 and 1110. However, students 1124 can still see and interact with students 1122 who are also in the physical classroom.

As another example, teacher 1110 may tell students to participate in group discussions, and students 1122 and 1124 may be sorted into the same group. In this example, the HMD may silence the virtual content and allow students 1112 and 1124 to participate in face-to-face discussions. The HMD can also reduce the size of the avatars 1110 and 1126 to reduce perceived confusion during group discussions.

Example of silencing a wearable in an entertainment context

Wearable systems can also detect trigger events and silence audio/visual content in the context of entertainment. For example, a wearable system could monitor a user's physiological data while the user is playing a game. If the physiological data indicates that the user is experiencing an agitated emotional state (such as being very angry due to a loss in a game or very scared during a game), the wearable system can detect the presence of a trigger event and thus can cause the HMD to automatically render the virtual content silence. Wearable systems can compare physiological data to one or more thresholds to detect triggering events. As an example, a wearable system may monitor the user's heart rate, respiration rate, pupil dilation, and the like. The threshold condition may depend on the type of game the user is playing. For example, if the user is playing a relatively relaxing game (such as a life simulation game), the threshold conditions (such as threshold heart rate, breathing rate, etc.) may be lower than if the user is playing a competitive game (which may require high concentration and may cause the user's heart rate rising). If the user's physiological state exceeds a threshold, the wearable system is triggered to silence the virtual content provided by the HMD.

As another example, virtual content may be associated with unpleasant music. Unpleasant music may be a trigger event for silencing the HMD's audio/visual content. Wearable systems can use inward-facing imaging systems (for example, to determine the user's facial expression or pupil dilation) or other environmental sensors (for example, to detect the user's breathing rate or heart rate) to detect user reactions. For example, a wearable system could detect that a user frowns while listening to certain music.

A wearable system can generate a warning message indicating that the user is experiencing an agitated emotional state. The HMD may display a virtual graphic advising the user to manually actuate the reality button 263 to silence the display of the virtual content. In some embodiments, the HMD may automatically turn off the virtual content if the HMD does not receive user confirmation within a certain period of time. The HMD can also automatically turn off virtual content in response to detecting a trigger event. For example, when unpleasant music is playing, the HMD can automatically silence the sound or lower the volume of the sound. At the same time, the HMD can still play the virtual image associated with the sound.

Example of silencing wearables in the context of shopping

Figure 1 IE shows an example of silencing an HMD in the context of shopping. In this example, a user 210 may be wearing an HMD in a shopping mall 1100e. User 210 can use the HMD to perceive virtual content such as her shopping list, price tags, recommended items (and their location in the store). Users may also perceive a physical booth 1150 with chefs 1152 selling various spices and cooking utensils.

The wearable system may detect the location of the user 210 using environmental sensors, such as GPS or outward-facing imaging systems. If the wearable system determines that the user 210 is within a threshold distance of the booth 1150, the HMD can automatically silence the display of the virtual content so that the user can interact with the chef 1152 in person. This may advantageously reduce perceived confusion when the user 210 is making conversions with the chef 1152 . Also, for example, a user may be able to tell which items in a booth are physical items (as opposed to virtual items). The wearable system may detect a termination condition, for example, when the user 210 exits the booth 1150, the HMD may unsilence the display of the virtual content in response to detecting the termination condition.

Example of silencing virtual content based on context

In addition to or instead of silencing virtual content based on events in the environment (e.g., an emergency) or objects in the environment (e.g., the presence of another user's face), the wearable system can also silence virtual content based on characteristics of the user's environment. Virtual content is silent. For example, the wearable system may recognize such characteristics of the user's environment based on objects observed by outward-facing imaging system 464 . Based on the type of user environment (e.g., home, office, lounge or gaming area, outdoors, retail store, mall, theater or concert venue, restaurant, museum, vehicle (e.g., car, airplane, bus, train, etc.), Wearable systems can customize virtual content or silence certain virtual content.

In addition to or instead of using the wearable system's outward-facing imaging system 464, as will be described further herein, the wearable system may use location sensors (e.g., GPS sensors) to determine the user's location and thus infer the user's location. the nature of the environment. For example, a wearable system may store locations of interest to the user (eg, home location, office location, etc.). Location sensors can determine location, compare to known locations of interest, and the wearable system can infer the user's environment (for example, the wearable system can determine that the user is in the home environment if the system's GPS coordinates are close enough to the user's home location and apply appropriate content blocking (or allowing) based on the home environment).

As one example, a wearable system may include various virtual content, such as virtual content related to social media, game invitations, audio-visual content, office content, and navigation applications. The outward-facing imaging system 464 may detect that the user is in the office (eg, by using an object recognizer to recognize the presence of a computer monitor, business phone, work files on a desk). Wearable systems can thus allow office applications and block social media feeds and game invites so users can focus on work. However, wearable systems can be configured not to silence navigation applications as they can help guide users to customer destinations. However, when the wearable system detects that the user is sitting on a chair in an office away from the user's desk (e.g., through analysis of images acquired by an outward-facing imaging system), the wearable system can be configured to allow social media feeds alone Or in combination with office (or navigation) apps, since the user might take a short break. Additionally or alternatively, the wearable system can flag the environment and specify what to block or allow based on user input. For example, the wearable system may receive an indication from the user that the scene is the user's bedroom, and the user may select the option to allow entertainment content or block work content at that scene. Thus, when the user re-enters the bedroom, the system can determine that the user is in the bedroom and automatically block or allow content based on user input.

In some cases, the wearable system may silence or present virtual content based on a combination of the environment and the user's role relative to the environment. For example, when the wearable system detects that the user is in an office (eg, by identifying office furniture using object recognizer 708), the wearable system may present the employee with a set of office tools and block access to the Internet (or other applications). However, if the executive enters the same office environment, the wearable system may allow the executive to access the Internet, since the executive may have more access to virtual content.

As another example, the wearable system may recognize that the user is in the house, such as by recognizing the presence of furniture (eg, sofa, television, dining table, etc.) in the environment or by manually marking it, such as by the user. Thus, wearable systems may allow certain virtual content, such as social media feeds, video games, or telepresence invitations from/to friends. In some implementations, even if two users are in the same environment, the virtual content perceived by the users may be different. For example, both the child and the parent can be in the living environment, but the wearable system can block virtual content that is inappropriate for the child's age, while allowing the parent to view such virtual content. Additional examples of silencing virtual content based on location are described further below with reference to FIGS. 11F and 11G .

Although examples are described with reference to blocking virtual content, the wearable system may also silence virtual content based on location, eg, by de-emphasizing some or all of the virtual content or turning off the display based on location.

Examples of Selective Content Silencing in a Work Environment

FIG. 11F shows an example of selectively blocking content in a work environment. Figure 1 IF shows two scenarios 1160a and 1160b, where some virtual content is blocked in scenario 1160b. Scenes 1160a and 1160b show office 1100f where user 210 is physically standing in the office. User 210 may wear HMD 1166 (which may be an embodiment of the HMD described with reference to FIG. 2 ). A user may perceive physical objects in the office through the HMD, such as a table 1164a, a chair 1164b, and a mirror 1164c. The HMD may also be configured to present virtual objects, such as virtual menus 1168 and virtual avatars 1164 for games.

In some cases, the wearable system may be configured to selectively silence virtual content in the user's environment such that not all of the virtual content is presented to the user by HMD 1166 . As one example, a wearable system may receive data about an environment obtained from one or more environmental sensors of the wearable system. The environmental data may include images of the office alone or in combination with GPS data. The environmental data may be used to identify objects in the user's environment or determine the user's location based on the identified objects. Referring to FIG. 11F , the wearable system can analyze environmental data to detect the physical presence of a workbench, chair 1164b, and mirror 1164c. Based at least in part on the received data detecting the workbench 1514, the chair 1512, and the mirror 1164c, the wearable system 200 may discern the environment as an office environment. For example, the wearable system may make this determination based on context information associated with the object, such as the characteristics of the object and the layout of the object. The collection of objects in the user's environment can also be used to determine the likelihood that the user is in a particular location. As an example, a wearable system may determine that the presence of an L-shaped desk and a rolling chair indicates a high likelihood that the environment is an office. Wearable systems can train and apply machine learning models (eg, neural networks) to determine the environment. Various machine learning algorithms (such as neural networks or supervised learning) can be trained and used to discriminate the environment. In various embodiments, one or more object discriminators 708 may be used for such discrimination. Alternatively, the user may have previously marked the location as "work" via user input.

Based on the environment, the wearable system can automatically block/not block (or allow/disallow) certain virtual content. The wearable system can access one or more settings associated with the environment to block virtual content. Referring to FIG. 11F , settings associated with an office environment may include muting video games. Thus, as shown in scene 1160b, the wearable system may automatically prevent avatar 1524 from being presented by HMD 1166 to allow user 210 to focus on his work. As another example, a wearable system may be configured to present an image of avatar 1164 but still prevent one or more user interfaces associated with avatar 1164 from operating. In this example, user 210 will still be able to see avatar 1164, but user 210 will not be able to interact with avatar 1164 while the wearable system enables settings associated with the work environment.

In some implementations, settings for silencing virtual content at a location may be user configurable. For example, a user may select which virtual content to block for an environment and which label to apply to that location and/or virtual content selection. Referring to FIG. 11F , when the user 210 is located in the office 1100f, the user 210 may choose to prevent the avatar 1164 from appearing in the HMD. The wearable system may then store settings associated with office 1100f and apply the settings to selectively block avatar 1164. Accordingly, avatar 1164 is blocked from the user's view, as shown in scene 1160b.

Examples are described with reference to determining the environment (e.g., an office) and silencing virtual content based on the environment, the wearable system may also silence virtual content (or components of the wearable system) based on environmental factors or the similarity of the content to other blocked content , so that the wearable system does not have to determine the specific location of the user. This may be advantageous if the wearable system does not include a location sensor, if the location sensor is blocked (eg, blocking the path to GPS satellites), or if the location accuracy is not sufficient to determine environmental characteristics. A wearable system can recognize objects in an environment and generally determine characteristics of an environment (eg, a leisure environment, a public environment, or a work environment) and silence virtual content based on the characteristics of the environment. For example, a wearable system can recognize that the user's environment includes sofas and televisions. Thus, the wearable system can determine that the user is in a leisure environment without knowing whether the leisure environment is actually the user's home or a break room at the user's work. In some implementations, the system will determine the type of environment and provide a notification to the user to accept or decline the environment label.

Example of selective content blocking in a lounge environment

FIG. 11G shows an example of selectively blocking content in a lounge environment. Figure 11G shows two scenarios 1170a and 1170b. Lounge 1100g shown in FIG. 11G shows user 210 wearing HMD 1166 and physically standing in lounge 1100g. Lounge 1100g includes physical objects such as table 1172c, sofa 1172b, and television 1172a. HMD 1166 may also be configured to present virtual content, such as virtual avatars 1176 and virtual menus 1174 for gaming, neither of which are physically present in the room. In this example, virtual menu 1174 presents user 210 with options 1178a, 1178b, 1178c to play a crossword, start a conference call, or access work email, respectively.

Wearable systems can be configured to silence some virtual content based on the user's environment. For example, outward-facing imaging system 464 may acquire images of the user's environment. The wearable system can analyze the images and detect the physical presence of the coffee table, sofa 1172b and television 1172a. Based at least in part on the presence of the coffee table, sofa 1172b, and television 1172a, wearable system 200 may then recognize that user 210 is in a lounge environment.

The wearable system may present or silence virtual content based on one or more settings associated with the user's environment. This setting may include silencing some virtual content in the environment or silencing a portion of the virtual content. As an example of silencing some virtual content, the wearable system may prevent virtual avatar 1176 from displaying while maintaining virtual menu 1174 . As an example of blocking a portion of virtual content, scene 1170b shows an example of blocking work-related content while the user is in a break room. As shown in scene 1170b, instead of blocking the entire virtual menu 1174, the wearable system can selectively block meeting option 1178b and work email option 1178c, but keep crossword option 1178a available for interaction since crossword option 1178a is entertainment related , while options 1178b and 1178c are work-related, and settings associated with the lounge environment enable blocking of work-related content. In some implementations, the meeting options 1178b and 1178c may still be visible to the user, but the wearable system may prevent the user from interacting with the options 1178b and 1178c while the user 210 is in the lounge 1100g.

In some implementations, the user can configure silence settings associated with the environment, and the wearable system can automatically block similar virtual content, even though certain portions of the virtual content may not be part of the silence settings. For example, a user may configure work settings for silencing a social networking application. The wearable system can automatically silence game invites, as both game invites and social networking apps are considered entertainment. As another example, a wearable system can be customized to present work email and office tools in an office environment. Based on this setup, the wearable system can also present work-related contacts for telepresence tools to customize virtual content to the office environment. The wearable system may use one or more machine learning algorithms described with reference to object discriminator 708 in FIG. 7 to determine whether virtual content is similar to those that are blocked (or customized).

Although the examples in scenarios 1170a and 1170b are described with reference to blocking content based on user context, in some implementations, settings associated with context may relate to allowing certain virtual content. For example, settings associated with a lounge environment may include enabling interaction with entertainment-related virtual content.

Example of trigger event

12A, 12B, and 12C illustrate examples of silencing virtual content presented by an HMD based at least in part on the occurrence of a trigger event. In Figure 12A, the user of the HMD can perceive a physical object in his FOV 1200a. Physical objects may include television (TV) 1210 , remote control 1212 , TV stand 1214 , and window 1216 . The HMD here may be an embodiment of the display 220 described with reference to FIGS. 2 and 4 . HMDs can display virtual objects on top of the user's physical environment in an AR or MR experience. For example, in FIG. 12A, the user may perceive virtual objects such as virtual building 1222 and avatar 1224 in the user's environment.

A user can interact with objects in the user's FOV. For example, an avatar 1224 may represent a virtual image of a friend of the user. When the user is having a telepresence session with his friend, the avatar 1224 can animate the actions and emotions of the user's friend to create a tangible (tangible) feeling of the friend's presence in the user's environment. As another example, the user may interact with the TV 1210 using the remote control 1212 or using a virtual remote control presented by the HMD. For example, a user may use the remote control 1212 or a virtual remote control to change channels, volume, sound settings, and the like. As yet another example, a user may interact with a virtual building 1222 . For example, a user may select virtual building 1222 using gestures (eg, hand gestures or other body gestures) or actuating a user input device (eg, user input device 504 in FIG. 4 ). Upon selection of the virtual building, the HMD may display a virtual environment inside the virtual building 1222. For example, virtual building 1222 may include a virtual classroom inside. In the AR/MR/VR environment, users can simulate walking into a virtual classroom and participating in courses.

When the user is in the AR/MR/VR environment, environmental sensors (including user sensors and external sensors) can acquire data on the user and the user's environment. A wearable system can analyze data acquired by environmental sensors to determine one or more triggering events. Upon the occurrence of a trigger event (which may have a magnitude or importance above a threshold), the wearable system may automatically silence the virtual content, for example by silencing the display of some or all of the visible virtual content or by silencing the audible virtual content silence.

Triggering events may be based on physical events occurring in the user's environment. For example, triggering events may include emergency or unsafe situations such as fires, ruptured arteries (during surgery), approaching police cars, chemical spills (during experiments or industrial processes), etc. Triggering events can also be associated with user actions, such as when the user is walking on a crowded street, sitting in a car (which may be unsafe to drive if too much virtual content is presented to the user). Triggering events can also be based on the user's location (eg, at home or in a park) or the context around the user (eg, work or leisure). Triggering events can also be based on objects in the user's environment, including other people. For example, the triggering event may be based on the density of people within a certain distance of the user or computer facial recognition of certain people (eg, teachers, police officers, supervisors, etc.) who have approached the user.

Additionally or alternatively, the triggering event may be based on virtual content. For example, a trigger event may include a sudden loud noise in an AR/VR/MR environment. Triggering events may also include unpleasant or disturbing experiences in AR/VR/MR environments. As yet another example, the wearable system may silence virtual content similar to virtual content that was previously blocked by the wearable system in a particular location.

Trigger events may also include a change in the user's location. FIG. 12D illustrates an example of silencing virtual content when a change in the user's environment is detected. In Figure 12D, user 210 is initially in lounge 1240b. A user may perceive virtual content customized for lounge 1240b through the HMD, such as example virtual content 1178a and 1176 shown in scene 1170b in FIG. 11G . User 210 may walk out of break room 1240b and into office 1240a. When user 210 transitions from lounge 1240b to office 1240a, wearable system 200 may acquire data from one or more environmental sensors. The acquired data may include images acquired by outward facing imaging system 464 . The wearable system can analyze the acquired images to detect the presence of the workbench 1242 , chair 1244 and computer monitor 1246 . Wearable system 200 may recognize that the user has entered the office environment based at least in part on the presence of one or more physical objects in the environment.

Because wearable system 200 detects that the environment has changed (eg, because the user walked from break room 1240b to office 1240a), wearable system 200 determines settings associated with silencing content for the new environment. For example, wearable system 200 may check whether a content blocking setting associated with office 1240a was previously enabled. If content blocking settings associated with office 1705 were previously enabled, wearable system 200 may automatically apply the relevant settings for content blocking. As an example, content blocking settings for office 1240a may include blocking entertainment content. Therefore, as shown in FIG. 12D, the user is no longer able to perceive the virtual game application. The wearable system may also remove the crossword application 1178a (perceivable by the user in the break room 1240b) and show the office tool application 1252 instead. As another example, the wearable system may update the contact list 1254 for the telepresence session to present work-related contacts (rather than the user's friends outside of work). The wearable system can also sort the contact list so that the user is more aware of work-related contacts when the user is in office 1240a (eg, move work-related contacts to the top of the contact list) .

Although in this example the user walks from the break room 1240b to the office 1240a, similar techniques can be applied if the user walks from the office 1240a to the break room 1240b. In some implementations, the wearable system can still apply the same settings to silence virtual content despite the user moving from one location to another because the scene has not changed. For example, a user may move from a park to a subway station. Wearable systems can apply the same settings to silence virtual content, since parks and subway stations can both be considered public scenes.

Detection of triggering events based on computer vision and sensors

Various techniques can be used to detect trigger events. The triggering event may be determined based on the user's reaction. For example, wearable systems can analyze data acquired by inward-facing imaging systems or physiological sensors. Wearable systems can use data to determine a user's emotional state. A wearable system can detect the presence of a trigger event by determining whether the user is in a certain emotional state (eg, angry, scared, uncomfortable, etc.). As an example, a wearable system may analyze the user's pupil dilation, heart rate, respiration rate, or sweat rate to determine the user's emotional state.

Computer vision techniques can also be used to detect triggering events. For example, a display system may analyze images acquired by an outward-facing imaging system to perform scene reconstruction, event detection, video tracking, object discrimination, object pose estimation, learning, indexing, motion estimation, or image restoration, among others. These tasks can be performed using one or more computer vision algorithms. Non-limiting examples of computer vision algorithms include: scale invariant feature transform (SIFT), accelerated robust feature (SURF), orientation (orient) FAST and rotate (rotate) BRIEF (ORB), binary robust Invariant Scalable Keypoint (BRISK), Fast Retinal Keypoint (FREAK), Viola-Jones Algorithm, Eigenfaces Method, Lucas-Kanade Algorithm, Horn-Schunk Algorithm, Mean-shift Algorithm, Visual Synchronous Localization and Mapping (vSLAM) Technology , sequential Bayesian estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative closest point (ICP), semi-global matching (SGM), semi-global block matching (SGBM), histogram of feature points, various machine learning algorithms (such as support vector machine, k-nearest neighbor algorithm, naive Bayesian, neural network (including convolutional or deep neural network) ), or other supervised/unsupervised models, etc.) and so on. As described with reference to FIG. 7, one or more of the computer vision algorithms may be implemented by object discriminator 708 for discerning objects, events, or environments.

One or more of these computer vision techniques may also be used with data acquired from other environmental sensors, such as, for example, microphones, to detect the presence of triggering events.

Triggering events may be detected based on one or more criteria. These criteria can be defined by the user. For example, a user may set the trigger event to be a fire in the user's environment. Thus, when the wearable system detects a fire using computer vision algorithms or using data received from smoke detectors (which may or may not be part of the wearable system), the wearable system can then signal the presence of the triggering event and automatically Silences the virtual content being displayed. Standards can also be set by another person. For example, a programmer of a wearable system can set the trigger event to be overheating of the wearable system.

The existence of a trigger event may also be indicated by user interaction. For example, a user may make a particular gesture (eg, a hand gesture or a body gesture) or actuate a user input device that indicates the presence of a triggering event.

Additionally or alternatively, criteria may also be learned based on the behavior of a user (or the behavior of a group of users). For example, a wearable system can monitor when the user turns off the HMD. The wearable system may observe that the user frequently turns off the wearable system in response to a certain type of virtual content (eg, certain types of scenes in a movie). The wearable system can learn the user's behavior accordingly and predict triggering events based on the user's behavior. As another example, a wearable system may correlate a user's emotional state based on the user's previous interactions with virtual content. Wearable systems can use this association to predict whether there is a trigger event when the user interacts with the virtual object.

Triggering events can also be based on known objects. For example, a wearable system could block virtual content from a display at a given location. The wearable system can automatically block other virtual content with similar characteristics for that given location. For example, a user can configure to block video viewing apps in the car. Based on this configuration, even if the user does not specifically configure the blocking of movie and music apps, the wearable system can automatically block movie and music apps, because movie and music apps have similar characteristics to video viewing apps (e.g., all of them are audio-visual entertainment content).

Machine learning that triggers events

Various machine learning algorithms can be used to learn triggering events. Once trained, wearable systems can store machine learning models for subsequent applications. As described with reference to FIG. 7 , one or more of the machine learning algorithms or models may be implemented by object discriminator 708 .

Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression), instance-based algorithms (e.g., learning vector quantization), decision tree algorithms (e.g., classification and regression trees), Bayesian algorithms (e.g., Naive Bayes), clustering algorithms (e.g., k-means clustering), association rule learning algorithms (e.g., a-priori algorithms), artificial neural network algorithms (e.g., perceptrons), deep learning algorithms (e.g., deep Boltzmann machines or deep neural networks), dimensionality reduction algorithms (e.g., principal component analysis), ensemble algorithms (e.g., stacked generalization), and/or other machine learning algorithm. In some embodiments, individual models can be customized for individual data sets. For example, a wearable device may generate or store base models. The base model can be used as a starting point to generate additional models specific to data types (eg, specific users), sets of data (eg, sets of additional images obtained), conditional situations, or other variants. In some embodiments, the wearable system can be configured to utilize a variety of techniques to generate models for analyzing aggregated data. Other techniques may include using pre-defined thresholds or data values.

Criteria can include threshold conditions. If analysis of data acquired by environmental sensors indicates that a threshold condition has been passed, the wearable system may detect the presence of a triggering event. Threshold conditions may involve quantitative and/or qualitative measurements. For example, a threshold condition may include a score or percentage associated with the likelihood of a triggering event occurring. A wearable system can compare a score calculated from data from environmental sensors to a threshold score. If the score is above a threshold level, the wearable system can detect the presence of a trigger event. In some embodiments, if the score is below a threshold, the wearable system may signal the presence of a triggering event.

Threshold conditions may also include letter grades such as "A", "B", "C", "D", etc. Each rating may represent the severity of the situation. For example, "A" might be the most severe, while "D" might be the least severe. When the wearable system determines that an event in the user's environment is sufficiently severe (compared to a threshold condition), the wearable system can indicate the presence of the triggering event and take action (eg, silence virtual content).

Threshold conditions may be determined based on objects (or people) in the user's physical environment. For example, threshold conditions may be determined based on the user's heart rate. If the user's heart rate exceeds a threshold amount (eg, a certain number of beats per minute), the wearable system can signal the presence of a triggering event. As another example described above with reference to FIGS. 11A and 11B , the user of the wearable system may be a surgeon operating on a patient. Threshold conditions may be based on the patient's blood loss, the patient's heart rate, or other physiological parameters. As described with reference to FIGS. 2 and 10 , the wearable system may acquire patient data from environmental sensors (eg, an outward-facing camera imaging the surgical site) or from external sources (eg, ECG data monitored by an electrocardiograph). As yet another example, threshold conditions may be determined based on the presence of certain objects in the user's environment, such as the presence of fire or smoke.

Threshold conditions may also be determined based on the virtual object being displayed to the user. As one example, a threshold condition may be based on the presence of a certain number of virtual objects (eg, multiple missed virtual telepresence calls from a person). As another example, threshold conditions may be based on user interactions with virtual objects. For example, the threshold condition may be the duration that a user watches a piece of virtual content.

In some embodiments, threshold conditions, machine learning algorithms, or computer vision algorithms may be specific to a particular context. For example, in the context of surgery, computer vision algorithms can be specialized to detect surgical events. As another example, a wearable system could implement a facial recognition algorithm (rather than an event tracking algorithm) in an educational context to detect whether a person is near the user.

example warning

The wearable system can provide an indication to the user of the presence of a triggering event. The indication may be in the form of a focus indicator. Focus indicators may include halos, colors, perceived size or depth changes (e.g., making virtual objects appear closer and/or larger when selected), changes in user interface elements (e.g., changing the shape of the cursor from The circle changes to an escalation mark), a message (with text or graphics), or other audible, tactile, or visual effect that draws the user's attention. Wearable systems can present focused indicators near the cause of the triggering event. For example, a user of a wearable system can cook on the stove and use the wearable system to watch a virtual TV show. However, a user may forget about the food he is cooking while watching a TV program. As a result, the food may burn, creating smoke or flames. Wearable systems can detect smoke or flames using environmental sensors or by analyzing images of the stove. The wearable system can further detect that the source of the smoke or flames is food on the stove. Thus, a wearable system could present a halo around food on the stove, indicating that it is burning. This implementation may be beneficial because the user may be able to cure the source of the triggering event (eg, by turning off the furnace) before the event escalates (eg, entering a house fire). When a trigger event occurs, the wearable system can automatically silence the display of virtual content (eg, a virtual TV show) that is not related to the trigger event, so that the user can focus on the trigger event. Continuing with the burning food example above, the wearable system could silence virtual content unrelated to food or the stove while emphasizing the source of the triggering event (eg, by continuing to display a halo around the burning food).

As another example, the focus indicator may be a warning message. For example, a warning message may include a brief description of the triggering event (eg, a fire on the second floor, a patient's blood loss exceeds a certain amount, etc.). In some embodiments, the warning message may also include one or more recommendations for remediating the triggering event. For example, a warning message could say to call firefighters, inject some type of blood, etc.

In some implementations, the wearable system can use the user's response to the warning message to update the wearable system's recognition of the triggering event. For example, a wearable system may recognize that a user has arrived at home based on images acquired by an outward-facing imaging system. Therefore, the wearable system can present virtual content customized to the user's home. But the user is actually at a friend's house. The user may provide instructions to dismiss virtual content or change settings, for example by actuating a real button, using hand gestures, or actuating a user input device. The wearable system can remember the user's response to the environment, and the next time the user is in the same house, the virtual content customized for the user's family will not be presented.

As another example, a wearable system may recognize an emergency situation and present a message to automatically turn off the display. Users can also provide instructions to prevent the wearable system from turning off the display. The wearable system may remember the user's response and use the response to update the model used by object recognizer 708 to determine the presence of an emergency condition.

Example of silencing parts of a wearable system or virtual content in response to a trigger event

In response to a trigger event, the wearable system can silence visual and audible virtual content. For example, the wearable system can automatically silence audio from the HMD, turn off virtual content displayed by the HMD, put the HMD into sleep mode, dim the HMD's light field, reduce the amount of virtual content (e.g., by hiding virtual content, turning virtual content out of FOV or reduce the size of virtual objects). In embodiments where the wearable system provides haptic virtual content (eg, vibration), the wearable system may additionally or alternatively silence the haptic virtual content. In addition to or instead of silencing audio or visual content, the wearable system may also silence one or more of the other components of the wearable system. For example, the wearable system may selectively suspend an outward-facing imaging system, an inward-facing imaging system, a microphone, or other sensitive sensors of the wearable system. For example, a wearable system may include two eye cameras configured to image a user's eyes. The wearable system can silence one or both eye cameras in response to a trigger event. As another example, a wearable system may turn off one or more cameras in an outward-facing imaging system configured to image the user's surroundings. In some embodiments, the wearable system may change one or more cameras in the inward-facing imaging system or the outward-facing imaging system to a low-resolution mode such that the captured images may not have fine details. These implementations can reduce battery consumption of wearable systems when the user is not viewing virtual content.

Continuing with the example user environments shown in FIGS. 12A-12C , FIG. 12B shows an example FOV with the wearable system's virtual display turned off. In this figure, the user can only perceive physical objects 1210, 1212, 1214, and 1216 in his FOV 1200b because the wearable system's virtual display has been turned off. This figure is the inverse of Figure 12A with the wearable system turned on. In FIG. 12A, the user can perceive the virtual objects 1222, 1224 in the FOV 1200a, while in FIG. 12B, the user cannot perceive the virtual objects 1222, 1224.

Advantageously, in some embodiments, the wearable system may allow for a faster restart or recovery after a triggering event by silencing the presentation of virtual content in response to the triggering event while keeping the rest of the wearable system components continuing to operate. . For example, a wearable system could silence (or turn off completely) a speaker or display while keeping the rest of the wearable system components running. Therefore, the wearable system may not require a complete restart of all components after the triggering event ceases, compared to a full restart if the wearable system is completely shut down. As one example, a wearable system may silence the display of a virtual image but keep the audio on. In this example, the wearable system can respond to a trigger event to reduce visual confusion while allowing the user to hear a warning through the wearable system's speaker. As another example, a trigger event may occur when a user is in a telepresence session. The wearable system can silence the virtual content and sounds associated with the telepresence session, but allow the telepresence application to run in the background of the wearable system. As yet another example, a wearable system can silence virtual content (and audio) while keeping one or more environmental sensors operating. In response to a trigger event, the wearable system may turn off the display while continuously acquiring data using a GPS sensor (for example). In this example, the wearable system could allow rescuers to more accurately pinpoint the user's location during an emergency.

12C shows an example FOV where the wearable system has reduced the amount of virtual content. The size of avatar 1224 in FOV 1200c has been reduced compared to FIG. 12A. Additionally, the wearable system has moved the avatar 1224 from near the center of the FOV to the bottom right corner. As a result, the avatar 1224 is de-emphasized and may create less perceptual confusion for the user. Additionally, the wearable system has moved the virtual building 1222 outside of the FOV 1200c. As a result, virtual object 1224 does not appear in FOV 1200c.

In addition to or instead of automatically silencing virtual content based on a trigger event, the wearable system may also silence virtual content when a user manually actuates a reality button (eg, reality button 263 in FIG. 2 ). For example, a user can press the reality button to turn off audio or visual content, or tap the reality button to move virtual content out of the FOV. Further details regarding the real button are described below with reference to FIGS. 14A and 14B.

In some embodiments, upon detection of a trigger event, the wearable system may present an audible, tactile, or visual indication of the trigger event to the user. If the user does not respond to the triggering event, the wearable system can automatically silence to reduce perception confusion. In other embodiments, the wearable system will be silent if the user responds to the indication of the triggering event. For example, the user may respond by actuating a real button or user input device, or by providing a gesture (eg, waving his hand in front of the outward-facing imaging system).

Example procedure for silencing a wearable

13A and 13B illustrate an example process for silencing a wearable system based on a trigger event. Processes 1310 and 1320 in Figures 13A and 13B (respectively) may be performed by the wearable systems described herein. In both procedures, one or more boxes may be optional or part of another box. In addition, these two processes do not need to be performed in the order shown by the arrows in the figure.

At block 1312 of process 1310, the wearable system may receive data from environmental sensors. Environmental sensors may include user sensors as well as external sensors. Accordingly, data acquired by environmental sensors may include data associated with the user and the user's physical environment. In some embodiments, the wearable system can communicate with another data source to obtain additional data. For example, a wearable system may communicate with a medical device to obtain patient data (eg, heart rate, breathing rate, disease history, etc.). As another example, the wearable system may communicate with a remote data store to determine information about the virtual objects the user is currently interacting with (eg, the type of movie the user is watching, previous interactions with the virtual objects, etc.). In some implementations, the wearable system can receive data from an external imaging system in communication with the wearable system or from an internal imaging system networked with the external imaging system.

At block 1314, the wearable system analyzes the data to detect trigger events. Wearable systems can analyze data in view of threshold conditions. If the data indicates that a threshold condition has been exceeded, the wearable system can detect the presence of a triggering event. Triggering events can be detected in real time using computer vision algorithms. Triggering events may also be detected based on one or more predictive models. For example, the wearable system may indicate the presence of a trigger event if the likelihood of the trigger event occurring exceeds a threshold condition.

At block 1316, the display system may be automatically silenced in response to the trigger event. For example, the wearable system may automatically turn off the display of virtual content or silence a portion of the virtual content presented by the display. As a result, users can see the physical environment through the wearable system without being disturbed by virtual content or having problems distinguishing real physical objects from virtual objects, or can perceive virtual content related to a particular environment. As another example, the wearable system may turn off or lower the volume of sounds associated with virtual content to reduce perceptual confusion.

At optional block 1318a, the wearable system may determine termination of the triggering event. For example, the wearable system can determine whether the situation that caused the triggering event is over (eg, a fire is extinguished) or the user is no longer in the same environment (eg, the user walks from home to the park). If the triggering event no longer exists, process 1310 may proceed to optional block 1318b to resume displaying the system or silent virtual content.

In some cases, the wearable system may determine the presence of the second triggering event at optional block 1318b. The second trigger event may cause the wearable system to resume displaying the system or a portion of the silenced virtual content, or cause the wearable system to silence other virtual content, the display system, or other components of the wearable system if they were not previously silenced.

Process 1320 in FIG. 13B illustrates another example process for silencing virtual content based on a triggering event. Blocks 1312 and 1314 in processes 1310 and 1320 follow the same description.

At block 1322 , the wearable system may determine whether a trigger event exists based on the analysis of the data at block 1314 . If there is no trigger event, process 1320 returns to block 1312, where the wearable system continues to monitor data acquired from environmental sensors.

If a trigger event is detected, at block 1324 the wearable system may provide an indication regarding the trigger event. As described with reference to FIG. 12A, the indication may be a focus indicator. For example, the indication may be a warning message. The warning message can indicate that a triggering event has been detected, and the wearable system can automatically silence the perceptual confusion if no response is received from the user for a period of time (eg, 5 seconds, 30 seconds, 1 minute, etc.).

At block 1324, the wearable system may determine whether a response to the indication has been received. The user may respond to the indication by actuating a user input device or a real button. Users can also respond by changing gestures. The wearable system can determine whether the user has provided a response by monitoring input from the user input device or real buttons. The wearable system can also analyze images acquired by the outward-facing imaging system or data acquired by the IMU to determine whether the user has changed his posture to provide a response.

If the wearable system does not receive a response, the wearable system may automatically silence the virtual content (or sound) at block 1328 . If the wearable system does receive a response, process 1320 ends. In some embodiments, if the wearable system receives a response, the wearable system may continue to monitor the environmental sensors. The wearable system can later detect another trigger event. In some embodiments, the response received from the user directs the wearable system to perform another action not provided in the instruction. As an example, the wearable system may provide a warning message indicating that the virtual display will be turned off after the user does not respond within a threshold duration. However, the user does respond within a duration, for example, by tapping twice on a real button. But this response is associated with dimming the light field (rather than turning it off). Therefore, the wearable system may instead dim the light field instead of turning it off as indicated in the warning message.

Process 1330 in Figure 13C illustrates an example of selectively blocking virtual content based on circumstances. Process 1330 may be performed by wearable system 200 as described herein.

Process 1330 begins at block 1332 and moves to block 1334 . At block 1334, the wearable system may receive data acquired from environmental sensors of the wearable device. For example, the wearable system may receive images acquired by an outward-facing imaging system 464 of the wearable device. In some implementations, the wearable system can receive data from an external imaging system in communication with the wearable system or from an internal imaging system networked with the external imaging system.

At block 1336, the wearable system analyzes the data collected and received by the environmental sensors. Based at least in part on data received from environmental sensors, the wearable system will discern the environment in which the user of the wearable system is currently located. As described with reference to FIG. 11F , the wearable system may discern the environment based on the presence of physical objects in the environment, the arrangement of physical objects in the environment, or the position of the user relative to the physical objects in the environment.

At block 1338, the wearable system checks the content blocking settings for the environment. For example, a wearable system can determine whether a user has entered a new environment (eg, whether the user has moved from a work environment to a leisure environment). If the wearable system determines that the user has not entered the new environment, the wearable system may apply the same settings as the previous environment, so blocks 1340-1352 may become optional.

At block 1340, the wearable system determines whether it has received an indication to enable or edit content blocking settings. Such indications may come from the user (eg, based on gestures by the user or input from a user input device). The indication can also be automatic. For example, a wearable system can automatically apply environment-specific settings in response to triggering events.

If the wearable system does not receive an indication, process 1330 moves to block 1350 where the wearable system determines whether content blocking settings have been previously enabled. If not, then at block 1352, the virtual content is presented without blocking. Otherwise, at block 1344, the wearable system may selectively block virtual content based on content blocking settings.

If the wearable system receives the indication, the wearable system can edit the content blocking settings or create new content blocking settings. In the event that settings need to be configured for a new environment, the wearable system may initiate storage of content blocking settings at block 1342 . Therefore, when the user re-enters the same or a similar new environment, the wearable system can automatically apply the content blocking settings. Additionally, if a user can reconfigure existing content blocking settings that will be stored and later applied to the same or similar circumstances.

The content blocking settings associated with the environment may reside locally on the wearable device (e.g., at the local processing and data module 260) or remotely in a network storage location accessible by a wired or wireless network (e.g., remotely data repository 280). In some embodiments, content blocking settings may reside partially on the wearable system and may reside partially in a network storage location accessible by a wired or wireless network.

At block 1344, the wearable system implements the stored content blocking settings associated with the new environment. By applying the content blocking settings associated with the new environment, some or all of the virtual content will be blocked according to the content blocking settings. The process then loops back to block 1332.

At block 1350, the wearable system may check whether the content blocking setting 1350 was previously enabled. If not, the wearable system may present the virtual content at block 1352 without blocking. Otherwise, the wearable system may selectively block virtual content at block 1344 based on content blocking settings. Blocks 1350-1352 and blocks 1340-1344 may run in parallel or sequentially. For example, the wearable system may check for previous content blocking settings while determining whether an indication to modify the content blocking settings for the environment has been received.

Manual Control of Wearable Display Systems

As described herein, embodiments of the wearable display system may automatically control the visual or audible display of virtual content based on the occurrence of triggering events in the user's environment. Additionally or alternatively, a user may wish to have the ability to manually silence visual or audible virtual content.

Accordingly, the display system may include a user-selectable display button 263 as described with reference to FIG. 2 . Reality button 263 may silence the wearable device's visual display 220 or audio system (e.g., speaker 240) in response to certain conditions, such as sudden loud noises, unpleasantness or unsafety in the physical or virtual environment experiences or situations, emergencies in the real world, or simply because the user wishes to experience more of an "actual" reality than augmented or mixed reality (e.g., talking to a friend without a display of virtual content).

Reality button 263 (once actuated) can cause the display system to turn off or dim the brightness of display 220 or audibly silence the audio from speaker 240 . As a result, user 210 will be able to more easily perceive physical objects in the environment because perceptual confusion caused by virtual objects or sound displays to the user will be reduced or eliminated. In some embodiments, when reality button 263 is actuated, the display system may continue to function normally while the rest of the display system (e.g., environmental sensors, user input devices, etc.) Restart) while turning off the VR or AR display 220 and speaker 600.

Reality button 263 may cause the display system to reduce the amount of virtual content. For example, the display system reduces the size of the virtual object in the FOV (eg, reduces the size of the avatar or another virtual object), makes the virtual object more transparent, or reduces the brightness of the displayed virtual object. Reality button 263 may additionally or alternatively cause the display system to move virtual content from one location to another, such as by moving a virtual object from inside the FOV to outside the FOV or from a central area to a peripheral area. Additionally or alternatively, reality button 263 may dim the light field generated by the display system, thus reducing the potential for perceptual confusion. In some implementations, the display system may silence only a portion of the virtual content when reality button 263 is actuated. For example, when a user of the wearable device is shopping in a store, the wearable device may display virtual content such as prices of clothes in the store and a map of a department store. In response to loud noises in the department store, after actuating the reality button 263, the wearable device can hide or move the virtual content related to the price of the clothes (e.g., move to the outside of the FOV), while the user needs to leave the store quickly. In this case, the wearable device leaves the map.

Reality button 263 may be a touch-sensitive sensor mounted to frame 230 of the display system or to a battery pack that provides power to the display system. The user may for example wear the battery pack on his waist. Reality button 263 may be a touch-sensitive area that a user may actuate, for example, by a touch gesture or by swiping along a track. For example, by swiping down on a touch-sensitive part, the wearable can be silenced, while by swiping up, the wearable can resume its normal functionality.

In some embodiments, the wearable device may (additionally or alternatively) include a virtual reality button that is not a physical button, but a function actuated by a user gesture. For example, a wearable device's outward-facing camera can image a user's hand gestures, and if it recognizes a specific "silence" gesture (for example, the user raises his hand and forms a fist), the wearable device will The displayed visual or auditory content is silent. In some embodiments, after the user actuates the display button 263, the display system may display a warning message 1430 (shown in FIG. 14A ) informing the user that the display will be silenced. In some embodiments, the display system will be silent after a period of time (e.g., 5 seconds as shown in FIG. Associated virtual button to unsilence. In other embodiments, the real button 263 must be actuated a second time or the virtual warning message 1430 must be actuated (or associated with the message 1430) before the display system silences the visual or audible display. This functionality may be beneficial in the event that the user inadvertently actuates the real button 263 but does not want the display system to go into silent mode.

After entering silent mode, the user can resume normal operation by actuating the reality button 263, accessing the user interface, speaking a command, or allowing a period of time to elapse.

14B is a flowchart illustrating an example process 1400 for manually actuating a silent mode of operation of a display system. Process 1400 may be performed by a display system. At block 1404, the process receives an indication that the reality button has been actuated. At optional block 1408, the process causes the display system to display a warning message indicating to the user that the display system will enter a silent mode of operation. In the silent mode of operation, the visual or audible display of virtual content may be muted. At optional decision block 1410, the process determines whether the user has provided an indication that the silent mode of operation should be canceled (eg, by the user actuating the reality button a second time or actuating a warning message). If a cancellation is received, the process ends. If no cancellation is received, in some implementations, the display system is visually or audibly silent after a period of time (eg, 3 seconds, 5 seconds, 10 seconds, etc.). Although the example process 1400 depicts receiving a cancellation request at block 1410 , in other embodiments, the process 1400 may determine whether an acknowledgment is received at block 1410 . If an acknowledgment is received, process 1400 moves to block 1412 and silences the display system, if no acknowledgment is received, process 1400 ends.

other aspects

In a first aspect, a head mounted device (HMD) configured to display augmented reality image content, the HMD comprising: a display configured to present virtual content, at least a portion of the display being transparent, and when At least a portion of the display is positioned in front of the user's eyes when the user is wearing the HMD such that the transparent portion transmits light from a portion of the environment in front of the user to the user's eyes to provide a view of the portion of the environment in front of the user, the display also configured to display virtual content to a user at multiple depth planes; an environmental sensor configured to acquire data associated with at least one of: (1) the user's environment, or (2) the user; and hardware processing The sensor is programmed to: receive data from the environmental sensors; analyze the data to detect a trigger event; in response to detecting the trigger event, provide an indication to the user that the trigger event occurred; and silence the display of the HMD.

In a second aspect, the HMD of aspect 1, wherein, to silence the display of the HMD, the hardware processor is at least programmed to: dim the light output by the display; turn off the display of the virtual content; reduce the size of the virtual content ; increasing the transparency of the virtual content; or changing the position of the virtual content presented by the display.

In aspect 3, the HMD according to any one of aspects 1-2, wherein the HMD further includes a speaker, and in order to silence the display of the HMD, the hardware processor is programmed to silence the speaker.

In a 4th aspect, the HMD according to any one of aspects 1-3, wherein, to analyze the data to detect the triggering event, the hardware processor is programmed to: analyze the data in view of a threshold condition associated with the presence of the triggering event ; If the threshold condition is exceeded, the presence of a trigger event is detected.

In a fifth aspect, the HMD according to any one of aspects 1-4, wherein the hardware processor is programmed with at least one of a machine learning algorithm or a computer vision algorithm to detect trigger events.

In a 6th aspect, the HMD according to any one of aspects 1-5, wherein the indication of the presence of the triggering event comprises a focus indicator associated with an element in the environment at least in part responsible for the triggering event .

In a seventh aspect, the HMD according to any one of aspects 1-6, wherein the indication of the presence of a trigger event includes a warning message, wherein the warning message indicates to the user at least one of the following: (1) the HMD will Automatically silence after time unless the user performs a cancel action, or (2) the HMD will not silence unless the user performs a confirm action.

In an eighth aspect, the HMD of aspect 7, wherein the canceling action or confirming action comprises at least one of: actuating a real button, actuating a virtual user interface element presented by a display, actuating a user input device, or detecting to the user's cancel or confirm gesture.

In aspect 9, the HMD according to any one of aspects 7-8, wherein in response to a user performing a cancel action, the hardware processor is programmed to unsilence the display or continue to display the virtual content.

In a tenth aspect, the HMD according to any one of aspects 7-9, wherein, in response to the user performing a confirmation action, the hardware processor is programmed to silence or stop displaying the virtual content.

In an eleventh aspect, the HMD of any one of aspects 1-10, wherein the environmental sensor comprises at least one of: a user sensor configured to measure data associated with a user of the HMD; or an external sensor , which is configured to measure data related to the user's environment.

In a 12th aspect, the HMD according to any one of aspects 1-11, wherein the triggering event comprises an emergency or unsafe condition in the user's environment.

In a 13th aspect, the HMD of any one of aspects 1-12, wherein the display comprises a light field display.

In a 14th aspect, the HMD according to any one of aspects 1-13, wherein the display comprises: a plurality of waveguides; one or more light sources configured to direct light into the plurality of waveguides.

In a fifteenth aspect, the HMD of aspect 14, wherein the one or more light sources comprise a fiber optic scanning projector.

In aspect 16, the HMD according to any one of aspects 1-15, wherein the environmental sensor comprises an outward-facing imaging system to image the user's environment; the data comprises an image of the environment acquired by the outward-facing imaging system and to analyze the data to detect trigger events, the hardware processor programmed to analyze the image of the environment through one or more of a neural network or computer vision algorithm.

In a seventeenth aspect, the HMD of aspect 16, wherein the neural network comprises a deep neural network or a convolutional neural network.

In aspect 18, the HMD according to any one of aspects 16-18, wherein the computer vision algorithm comprises one or more of: scale invariant feature transform (SIFT), accelerated robust feature (SURF), Orientation FAST and Rotation BRIEF (ORB), Binary Robust Invariant Scalable Keypoint (BRISK) Algorithm, Fast Retinal Keypoint (FREAK) Algorithm, Viola-Jones Algorithm, Eigenface Algorithm, Lucas-Kannard Algorithm , Horn-Schenk algorithm, mean shift algorithm, visual simultaneous localization and mapping (vSLAM) algorithm, sequential Bayesian estimator, Kalman filter, bundle adjustment algorithm, self-adjusting threshold algorithm, iterative closest point (ICP) algorithm, semi-global matching (SGM) algorithm, semi-global block matching (SGBM) algorithm, feature point histogram algorithm, support vector machine, k-nearest neighbor algorithm or Bayesian model.

In a nineteenth aspect, the HMD according to any one of aspects 1-18, wherein the environmental sensor comprises an outward-facing imaging system imaging the user's environment; the data comprises an image of the environment acquired by the outward-facing imaging system; and to analyze the data to detect the triggering event, the hardware processor is programmed to: access a first image of the environment; access a second image of the environment, the second image being acquired by the outward facing imaging system after the first image; The second image is compared with the first image to determine the occurrence of a trigger event.

In aspect 20, the HMD of any one of aspects 1-19, wherein the environmental sensor comprises an outward-facing imaging system that images the user's environment, including the surgical site; Images of the surgical site are acquired; and to analyze the data to detect trigger events, the hardware processor is programmed to: monitor a medical condition occurring in the surgical site; detect a change in the medical condition; determine that the change in the medical condition exceeds a threshold.

In a twenty-first aspect, an HMD configured to display augmented reality image content, the HMD comprising: a display configured to present virtual content, at least a portion of the display is transparent, and the display is configured to display when the HMD is worn by a user At least a portion of the display is disposed at a position in front of the user's eyes such that the transparent portion transmits light from a portion of the environment in front of the user to the user's eyes to provide a view of the portion of the environment in front of the user. virtual content is displayed to a user at the depth plane; a user-actuatable button; and a hardware processor programmed to: receive an indication that the user-actuatable button has been actuated; and silence a display of the HMD in response to the indication.

In a 22nd aspect, the HMD of aspect 21, wherein, to silence the display of the HMD, the hardware processor is programmed at least to: dim the light output by the display; turn off the display of the virtual content; reduce the size of the virtual content ; increasing the transparency of the virtual content; or changing the position of the virtual content presented by the display.

In aspect 23, the HMD of aspect 21 or aspect 22, wherein the HMD further comprises a speaker, and in order to silence the display of the HMD, the hardware processor is programmed to silence the speaker.

In a 24th aspect, the HMD of any one of aspects 21-23, wherein, in response to the indication, the hardware processor is programmed to provide a warning to the user.

In a 25th aspect, the HMD of aspect 24, wherein the warning comprises a visual warning presented by a display or an audible warning provided by a speaker.

In aspect 26, the HMD of any of aspects 24-25, wherein the warning indicates to the user at least one of: (1) the HMD will automatically silence after a period of time unless the user takes a cancel action, or (2) The HMD will not be silent unless the user performs a confirmation action.

In a 27th aspect, the HMD of aspect 26, wherein the canceling action or confirming action comprises at least one of: actuating a user actuatable button, actuating a virtual user interface element presented by the display, actuating a user input The device or detects a cancel or confirm gesture from the user.

In aspect 28, the HMD according to any one of aspects 26-27, wherein in response to the user performing a cancel action, the hardware processor is programmed to unsilence the display or continue to display the virtual content.

In aspect 29, the HMD of any one of aspects 26-28, wherein the hardware processor is programmed to silence or stop displaying the virtual content in response to the user performing a confirmation action.

In a 30th aspect, the HMD of any one of aspects 21-29, wherein the hardware processor is further programmed to: receive a second indication that the user-actuatable button has been actuated; and in response to the second indication, cause the HMD to The display cancels the silence.

In aspect 31, a wearable system configured to display virtual content in a mixed reality or virtual reality environment, the wearable system comprising: a display configured to display virtual content in a mixed reality, augmented reality or virtual reality environment presenting virtual content; and a hardware processor programmed to: receive an image of the user's environment; analyze the image using one or more object recognizers configured to recognize objects in the environment using a machine learning algorithm The subject; detecting a triggering event based at least in part on the analysis of the image; in response to detecting the triggering event: in response to determining that a threshold condition associated with the triggering event is met, silencing the display.

In a 32nd aspect, the wearable system of aspect 31, wherein, to silence the display, the hardware processor is programmed to at least: dim the light output by the display; turn off the display of the virtual content; reduce the size of the virtual content ; increasing the transparency of the virtual content; or changing the position of the virtual content presented by the display.

In aspect 33, the wearable system of any one of aspects 31-33, wherein the hardware processor is further programmed to: detect a termination condition of the trigger event; and resume display in response to detecting the termination condition.

In a 34th aspect, the wearable system of aspect 33, wherein, to detect the termination condition, the wearable system is programmed to: determine whether the triggering event has terminated; or determine whether the user has left the environment in which the triggering event occurred.

In aspect 35, the wearable system of any one of aspects 31-34, wherein the hardware processing is further programmed to silence a speaker of the wearable system in response to detecting a trigger event.

In aspect 36, the wearable system of any one of aspects 31-35, wherein, in response to a trigger event, the hardware processor is further programmed to provide an indication of the presence of the trigger event, wherein the indication comprises at least one of: a focus indicator associated with an element in the environment that is at least partially responsible for triggering the event; or a warning message, wherein the warning message indicates to the user at least one of the following: (1) the HMD will Automatically silence after a period of time unless the user performs a cancel action, or (2) the HMD will not silence unless the user performs a confirm action.

In a 37th aspect, the wearable system of aspect 36, wherein the threshold condition associated with the triggering event includes a duration of time for which a cancel action is not detected.

In aspect 38, the wearable system of aspect 36 or 37, wherein the cancel action or confirm action comprises at least one of: actuating a real button, actuating a virtual user interface element presented by the display, actuating a user The input device or detection of a cancel or confirm gesture by the user.

In aspect 39, the wearable system according to any one of aspects 31-38, wherein the trigger event comprises an emergency or unsafe condition in the user's environment.

In aspect 40, the wearable system according to any one of aspects 31-39, wherein the machine learning algorithm comprises a deep neural network or a convolutional neural network.

In a 41st aspect, a method for displaying virtual content in a mixed reality or virtual reality environment, the method comprising: receiving an image of a user's environment; analyzing the image using one or more object discriminators, the one or more objects The discriminator is configured to discern objects in the environment; detect a trigger event based at least in part on the analysis of the image; in response to detecting the trigger event: in response to determining that a threshold condition associated with the trigger event is met, silence the virtual content. The method can be performed under the control of a hardware processor. A hardware processor may be provided in the augmented reality display device.

In aspect 42, the method of aspect 41, wherein silencing the virtual content comprises at least one of: preventing the virtual content from being presented; disabling interaction with the virtual content; turning off display of the virtual content; size; increase the transparency of the virtual content; or change the position of the virtual content rendered by the display.

In aspect 43, the method according to any one of aspects 41-42, further comprising: detecting a termination condition of the trigger event; and resuming display in response to detecting the termination condition.

In aspect 44, the method of aspect 43, wherein, to detect the termination condition, the wearable system is programmed to: determine whether the triggering event has terminated; or determine whether the user has left the environment in which the triggering event occurred.

In aspect 45, the method of any one of aspects 41-44, wherein analyzing the image includes identifying objects in the user's environment; and determining a trigger event includes determining a location of the user based at least in part on the identified objects.

In aspect 46, the method of aspect 45, wherein the triggering event comprises a change in the location of the user or a change in the scene around the user.

In aspect 47, the method of aspect 45 or 46, wherein, in response to detecting the triggering event, the method further comprises: accessing settings for silencing the virtual content at the location, and silencing the virtual content according to the settings .

In aspect 48, the method of any one of aspects 45-47, wherein recognizing objects in the user's environment is performed by a neural network.

In aspect 49, the method according to any one of aspects 41-48, wherein the threshold condition associated with the triggering event comprises a duration for which no cancellation action has been detected.

In aspect 50, the method according to any one of aspects 41-49, wherein the canceling action comprises at least one of: actuating a real button, actuating a virtual user interface element presented by a display, actuating a user input The device or detects a cancel or confirm gesture from the user.

other considerations

Each of the processes, methods, and algorithms described herein and/or depicted in the accompanying drawings can be embodied in and automated, in whole or in part, by a code module that is implemented by one or more physical computing systems, hardware Computer processors, application-specific circuit implementations; and/or electronic hardware configured to carry out specific and specific computer instructions. For example, a computing system can include a general purpose computer (eg, a server), special purpose circuitry, etc. programmed with specific computer instructions or a special purpose computer. Code modules can be compiled and linked into an executable program, installed in a dynamically linked library, or written in an interpreted programming language. In some implementations, certain operations and methods may be performed by circuitry specific to a given function.

Furthermore, particular implementations of the disclosed functions are sufficiently mathematically, computationally, or technically complex that in order to perform the described functions (eg, due to the amount of computation or complexity involved) or to provide results in substantially real-time, Dedicated hardware or one or more physical computing devices (with appropriate proprietary executable instructions) may be necessary. For example, animation or video may include multiple frames, each frame having millions of pixels, and specially programmed computer hardware is necessary in order to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

Modules of code or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer memory, including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), Optical discs, volatile or non-volatile memory, and combinations of the same and/or similar elements. Methods and modules (or data) may also be transmitted as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagated signal) over a variety of computer-readable transmission media, including wireless-based media and Wire/cable based medium and can take various forms (eg, as part of a single or multiplexed analog signal, or as multiple discrete digital data packets or frames). The results of disclosed procedures or processing steps may be stored persistently or otherwise in any type of non-transitory tangible computer memory, or may be transmitted via a computer-readable transmission medium.

Any process, block, state, step or function in the flowcharts described herein and/or depicted in the accompanying drawings should be understood as potentially representing a code module, code segment, or code portion, which is included in the process to achieve a specific function ( One or more of such as logical functions or arithmetic functions) or steps may perform the instructions. Various processes, blocks, states, steps, or functions can be combined, rearranged, added, deleted, modified, or otherwise changed according to the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functions described herein. Nor are the methods and processes described herein limited to any particular order, and blocks, steps or states related thereto can be performed in any other order as appropriate, such as in series, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed example embodiments. Furthermore, the separation of various system components in the embodiments described herein is for purposes of illustration and should not be construed as requiring such separation in all embodiments. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variants are possible.

The processes, methods and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

The systems and methods of the present disclosure each have several innovative aspects, no single one of which is solely responsible for, or required for, the desirability disclosed herein. The various features and processes described above can be used independently of each other or can be used in various combinations. All possible combinations and subcombinations are intended to fall within the scope of this disclosure. Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the spirit or scope of this disclosure. Way. Thus, the claims are not intended to be limited to the implementations shown herein but are to be accorded the widest scope consistent with the disclosure, principles and novel features disclosed herein.

Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Furthermore, although features above may be described as being performed in certain combinations, and even initially claimed as such, in some cases one or more features from a claimed combination can be deleted from that combination and all A claimed combination may relate to a subcombination or a variation of a subcombination. No single feature or group of features is required or integral to every embodiment.

As used herein, conditional words such as (among others) "could," "could," "may," "may," "for example," etc. are generally intended to mean that certain embodiments include and other embodiments do not include certain These features, elements and/or steps are unless otherwise specifically stated or otherwise understood in context. Thus, such conditionals are generally not intended to imply that the features, elements and/or steps are in any way essential to one or more embodiments, or that one or more embodiments necessarily include The logic that determines whether such features, elements and/or steps are included or will be implemented in any particular embodiment is entered or prompted. The terms "comprising", "comprising", "having", etc. are synonyms and are used in an open-ended manner inclusively and do not exclude additional elements, features, acts, operations, etc. Furthermore, the term "or" is used in its inclusive sense rather than its exclusive sense, thus, when used, for example, to concatenate a list of elements, the term "or" means one, some or all of the elements in the list. Additionally, the articles "a," "an," and "the" as used in this application and the appended claims should be construed to mean "one or more" or "at least one" unless specifically stated otherwise. .

As used herein, the phrase "at least one" referring to a list of items refers to any combination of those items, including individual members. As an example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B, and C. Connectives such as the phrase "at least one of X, Y, and Z" (unless stated otherwise) are understood in their commonly used context to express that an item, term, etc. may be at least one of X, Y, or Z. Thus, such conjunctions are generally not intended to imply that certain embodiments require the presence of each of at least one of X, at least one of Y, and at least one of Z.

Similarly, while operations may be depicted in the figures in the particular order, it should be appreciated that such operations need not be performed in the particular order, or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Additionally, the drawings may schematically depict one or more example processes in flowchart form. However, other operations not shown can be incorporated into the schematically shown example methods and processes. For example, one or more additional operations can be performed before, after, concurrently with, or during any illustrated operation. Additionally, operations may be rearranged or reordered in other implementations. In some cases, multitasking and parallel processing can have advantages. Furthermore, the separation of various system components described in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems can generally be integrated together in a single software product or Packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

Claims (14)

1. A wearable system configured to display virtual content in a mixed reality or virtual reality environment, the wearable system comprising:

a display configured to present virtual content in a mixed reality, augmented reality, or virtual reality environment; and

a hardware processor programmed to:

receiving an image of a physical environment of a user of the wearable system;

analyzing the image using one or more object discriminators configured to discriminate physical objects in the physical environment using a machine learning algorithm;

detecting a trigger event based at least in part on the analysis of the image, the trigger event comprising movement of the user between a first physical environment and a second physical environment, wherein the movement is determined based at least in part on: detecting at least one physical object present in the second physical environment but not in the first physical environment based on the analysis of the one or more object discriminators;

in response to detecting the trigger event:

accessing content blocking settings associated with the second physical environment;

determining one or more virtual content items available for silencing in the second physical environment based on the content blocking settings associated with the second physical environment; and

The determined one or more virtual content items are silenced.

2. The wearable system of claim 1, wherein to silence the determined one or more virtual content items, the hardware processor is programmed to at least:

dimming the light output by the display;

closing the display of the virtual content;

reducing the size of the virtual content;

increasing transparency of the virtual content; or alternatively

Changing the position of the virtual content presented by the display.

3. The wearable system of claim 1, wherein the hardware processor is further programmed to:

detecting a termination condition of the trigger event; and

in response to detecting the termination condition, ceasing to silence the determined one or more virtual content items.

4. A wearable system according to claim 3, wherein to detect the termination condition, the wearable system is programmed to:

determining whether the user has left the second physical environment in which the silencing of the determined one or more virtual content items occurred.

5. The wearable system of claim 1, wherein the hardware process is further programmed to silence a speaker of the wearable system in response to detecting the trigger event.

6. The wearable system of claim 1, wherein, in response to the detected trigger event, the hardware processor is further programmed to provide an indication of the presence of the trigger event, wherein the indication comprises at least one of:

a focus indicator associated with an element in the environment, the focus indicator being at least partially responsible for the trigger event; or alternatively

A warning message, wherein the warning message indicates to the user at least one of: (1) The determined one or more virtual content items will automatically silence after a period of time unless the user performs a cancel action, or (2) the determined one or more virtual content items will not silence unless the user performs a confirm action.

7. The wearable system of claim 6, wherein the cancel action or the confirm action comprises at least one of: actuating a real button, actuating a virtual user interface element presented by the display, actuating a user input device, or detecting a cancel or confirm gesture of the user.

8. The wearable system of claim 1, wherein the machine learning algorithm comprises a deep neural network or a convolutional neural network.

9. A method for displaying virtual content in a mixed reality or virtual reality environment, the method comprising:

under control of a hardware processor:

receiving an image of a physical environment of a user of the wearable system;

analyzing the image using one or more object discriminators configured to discriminate physical objects in the physical environment;

detecting a trigger event based at least in part on the analysis of the image, the trigger event comprising movement of the user between a first physical environment and a second physical environment, wherein the movement is determined based at least in part on: detecting at least one physical object present in the second physical environment but not in the first physical environment based on the analysis of the one or more object discriminators;

in response to detecting the trigger event:

accessing content blocking settings associated with the second physical environment;

determining one or more virtual content items available for silencing in the second physical environment based on the content blocking settings associated with the second physical environment; and

the determined one or more virtual content items are silenced.

10. The method of claim 9, wherein silencing the virtual content comprises at least one of:

preventing the virtual content from being presented;

disabling interaction with the virtual content;

closing the display of the virtual content;

reducing the size of the virtual content;

the transparency of the virtual content is improved; or alternatively

Changing the position of the virtual content presented by the display.

11. The method of claim 9, further comprising:

detecting a termination condition of the trigger event; and

the display is restored in response to detecting the termination condition.

12. The method of claim 11, wherein to detect the termination condition, the hardware processor is programmed to:

a determination is made as to whether the user has left the second physical environment.

13. The method of claim 9, wherein analyzing the image comprises discriminating between objects in the first physical environment; and detecting the trigger event includes determining a location of the user based at least in part on the recognized object.

14. The method of claim 13, wherein discriminating the object in the first physical environment is performed by a neural network.