JP6705124B2 - Head-mounted display device, information system, head-mounted display device control method, and computer program - Google Patents

Description

The present invention relates to a head-mounted display device.

A head-mounted display device (head mounted display, HMD) that is a display device mounted on the head is known. A head-mounted display device is used, for example, by generating image light using a liquid crystal display and a light source and guiding the generated image light to a user's eye using a projection optical system or a light guide plate. Make the person visually recognize the virtual image. There are two types of head-mounted display devices, a transmissive type in which a user can visually recognize an outside scene in addition to a virtual image and a non-transmissive type in which a user cannot visually recognize an outside scene. The transmissive head-mounted display device includes an optical transmissive type and a video transmissive type.

Patent Document 1 discloses an HMD that displays an advertisement associated with a specific object related to the position of the user of the HMD as an image of augmented reality.

Special table 2013-521576 gazette

However, in the technique described in Patent Document 1, an advertisement associated with a specific object that is not visually recognized by the user is not displayed, or an advertisement that is visually recognized by the user but is unnecessary for the user is displayed. In some cases, there is room for improving the convenience of the user. In addition, in the conventional HMD, downsizing of the HMD has been desired.

The present invention has been made to solve at least a part of the problems described above, and can be realized as the following modes.
[Form 1]
A transmissive head-mounted display device,
An image display unit that can display an image and can transmit the outside scene,
A position information acquisition unit that acquires position information of the head-mounted display device,
A determination information acquisition unit that acquires an image of a signboard as determination information used to determine whether to display an image displayed on the image display unit,
An information transmitting/receiving unit that is associated with the acquired position information and receives display information including determination information corresponding information,
When the determination information is acquired, a display for displaying the image as the determination information corresponding information included in the received display information, which is associated with the acquired determination information, on the image display unit. An image control unit,
A personal information storage unit for storing personal information;
The display image control unit causes the image display unit to display an image as the determination information corresponding information set based on the personal information of the user,
The personal information includes a central average time as a central average value when the user stayed in the past in each specific place,
The display image control unit, the time difference between the central average time and the current time is smaller than the threshold value, corresponding to the signboard of the image corresponding to specific locations before SL corresponding to the case where the small A head-mounted display device for displaying the image as judgment information corresponding information .

(1) According to one aspect of the invention, a transmissive head-mounted display device is provided. This head-mounted display device displays an image and an image display unit capable of transmitting an outside scene; a position information acquisition unit that acquires position information of the head-mounted display device; A determination information acquisition unit that acquires determination information used to determine whether or not to display an image to be displayed; an information transmission/reception unit that receives display information including determination information corresponding information associated with the acquired position information. And; when the judgment information is acquired, an image as the judgment information corresponding information included in the received display information that is associated with the acquired judgment information is displayed on the image display unit. And a display image control unit. According to the head-mounted display device of this aspect, the user can visually recognize the image as the determination information corresponding information based on the display information based on the specified position and the determination information, and the image unnecessary for the user is displayed. Since it is not displayed on the image display unit, the convenience for the user is improved.

(2) In the head-mounted display device according to the above aspect, the determination information acquisition unit includes an image detection unit that detects an identification target image as the determination information included in the display information from the outside scene. The display image control unit may display the determination information corresponding information on the image display unit when the identification target image is detected. According to the head-mounted display device of this aspect, since the user of the head-mounted display device can visually recognize the identification target image and can visually recognize the image as the determination information corresponding information associated with the identification target image, User convenience is improved.

(3) In the head-mounted display device of the above aspect, the image detection unit images the outside scene; and the display image control unit detects the identification target image in the taken outside scene. Alternatively, the determination information correspondence information associated with the detected identification target image may be displayed on the image display unit. According to the head-mounted display device of this aspect, since the image as the determination information corresponding information associated with the identification target image that is not imaged is not displayed, the user associates only with the signboard visually recognized by the user. The image as the determined determination information corresponding information can be visually recognized, and the unnecessary image as the determination information corresponding information does not have to be visually recognized, so that the convenience of the user can be improved.

(4) In the head-mounted display device of the above aspect, the image detection unit identifies the orientation of the image display unit; the display image control unit, based on the identified orientation of the image display unit, When the identification target image is detected in the outside scene, the determination information corresponding information associated with the detected identification target image may be displayed on the image display unit. According to the head-mounted display device of this aspect, the user can visually recognize the image as the determination information corresponding information associated with the identification target image in the line-of-sight direction of the user, and the identification that does not exist in the line-of-sight direction. It is not necessary to visually recognize the image as the unnecessary determination information corresponding information associated with the target image, and the convenience of the user is further improved.

(5) In the head-mounted display device according to the above-described aspect, further: a position of the head-mounted display device specified by the acquired position information and a signboard as a base of the detected identification target image. A distance measuring unit that measures the distance between the position and the position; the display image control unit sets the display mode of the image as the determination information corresponding information displayed on the image display unit based on the measured distance. You may. According to the head-mounted display device of this aspect, the user of the head-mounted display device, based on the image as the judgment information corresponding information displayed on the image display unit, in addition to the position of the actual signboard to be visually recognized. Since the position of the signboard can be recognized, the convenience for the user is further improved.

(6) In the head-mounted display device according to the above aspect, the display image control unit has at least an image size and a display position as a display mode of the image as the determination information corresponding information displayed on the image display unit. One may be set. According to the head-mounted display device of this aspect, since the user of the head-mounted display device can recognize the distance from the position of the signboard by the size or display position of the image as the determination information corresponding information, It is easy to recognize the distance from the signboard position.

(7) In the head-mounted display device of the above-described aspect, further comprising: a state detection unit that detects a state of the external environment of the head-mounted display device; and the display image control unit, the detected external environment. The image as the determination information corresponding information associated with the state of may be displayed on the image display unit. According to the head-mounted display device of this aspect, the image as the determination information corresponding information in consideration of the external environment of the head-mounted display device is displayed on the image display unit, so that the user can Since the information suitable for the situation can be recognized, the convenience for the user is further improved.

(8) In the head-mounted display device according to the above aspect, a specific image is further displayed on the image display unit, an image of a specific object in the outside scene is detected, and the specific image and the specific image are detected. A calibration unit that calibrates the positional relationship between the image displayed on the image display unit and the target object based on the image of the target object; and the display image control unit is calibrated by the calibration unit. The image as the determination information corresponding information displayed on the image display unit may be changed based on the position of the image of the specific object detected in the outside scene. According to the head-mounted display device of this aspect, by superimposing the actual specific object and the specific image displayed on the image display unit, the user can display the specific object and the specific image. Can be recognized as an object in the same space, and the usability in user operation is improved.

(9) In the head-mounted display device of the above-described mode, further comprising: a personal information storage unit for storing personal information; the display image control unit, the judgment information corresponding information set based on the personal information of the user. The image may be displayed on the image display unit. According to the head-mounted display device of this aspect, the image as the determination information corresponding information in which the personal information of the user of the head-mounted display device is also considered is displayed on the image display unit. Since the information that suits one's taste can be recognized, the convenience of the user is further improved.

(10) In the head-mounted display device of the above aspect, the personal information of the user may be information about a place where the user has stayed in the past. According to the head-mounted display device of this aspect, an image in which the place where the user of the head-mounted display device has stayed in the past is taken into consideration is displayed on the image display unit. Since the information of the high place can be recognized with priority over the information of other places, the convenience of the user is further improved.

(11) In the head-mounted display device according to the above aspect, the determination information acquisition unit may use, as the determination information, information about a place where the user has stayed in the past, biometric information of the user, and other users. And at least one of the statistical information of 1. According to the head-mounted display device of this aspect, the user can visually recognize the image as the determination information corresponding information based on the position information and the determination information other than the image without detecting the identification target image. The convenience of the user is further improved.

(12) According to another aspect of the invention, a transmissive head-mounted display device is provided. This head-mounted display device displays an image and an image display unit capable of transmitting an outside scene; a position information acquisition unit that acquires position information of the head-mounted display device; An advertisement information storage unit that stores display information including an image as judgment information corresponding information; and a judgment information acquisition unit that acquires judgment information used for determining whether or not to display an image as the judgment information corresponding information. A display image control unit that causes the image display unit to display an image as the determination information correspondence information associated with the acquired determination information when the determination information is acquired. According to the head-mounted display device of this aspect, it is not necessary to transmit/receive information to/from another information storage device, so that the convenience of the user is improved and the head-mounted display device is downsized. it can.

(13) According to another aspect of the present invention, a transmissive head-mounted display device is provided. This head-mounted display device includes an image display section that displays an image and is capable of transmitting an outside scene; determination information that acquires determination information used to determine whether or not to display an image displayed on the image display section. An acquisition unit; an information transmission/reception unit that receives display information including determination information corresponding information; and, if the determination information is acquired, it is obtained from the determination information corresponding information included in the display information. A display image control unit that causes the image display unit to display an image as the determination information corresponding information associated with the determination information. According to the head-mounted display device of this aspect, even if there is no position information of the head-mounted display device, based on the acquired determination target and the received display information, as the determination information corresponding information. Since the image is displayed, the convenience for the user can be improved.

All of the plurality of constituent elements of each mode of the present invention described above are not indispensable, and in order to solve part or all of the above problems, or part or all of the effects described in the present specification. In order to achieve the above, it is possible to appropriately change or delete some of the constituent elements of the plurality of constituent elements, replace them with new constituent elements, and partially delete the limited contents. Further, in order to solve some or all of the above problems, or to achieve some or all of the effects described in the present specification, the technical features included in one embodiment of the present invention described above. It is also possible to combine a part or all of the above-mentioned technical features included in the other forms of the present invention with a part or all of them to form an independent form of the present invention.

For example, according to one aspect of the present invention, one or more of five elements of an image display unit, a position specifying unit, a determination information acquisition unit, an information transmission/reception unit, and a display image control unit are provided. It is feasible as a device equipped with it. That is, this device may or may not have the image display unit. Further, the device may or may not include the determination information acquisition unit. The device may or may not include the position information acquisition unit. The device may or may not include the information transmitting/receiving unit. The device may or may not include the display image control unit. The image display unit may be capable of displaying an image and transmitting an outside scene, for example. The position information acquisition unit may acquire position information of the head-mounted display device, for example. The judgment information acquisition unit may acquire, for example, judgment information used for determining whether or not to display the image displayed on the image display unit. The information transmission/reception unit may receive the display information, which is associated with the information on the identified position and includes the determination information corresponding information, for example. The display image control unit, for example, when the judgment information is acquired, is associated with the acquired judgment information, the image as the judgment information corresponding information included in the received display information, It may be displayed on the display unit. Such a device can be realized as, for example, a head-mounted display device, but can also be realized as a device other than the head-mounted display device. According to such a form, at least one of various problems such as improvement and simplification of operability of the device, integration of the device, and improvement of convenience for a user who uses the device can be solved. it can. Any or all of the technical features of the respective forms of the head-mounted display device described above can be applied to this device.

The present invention can be implemented in various forms other than the head-mounted display device. For example, a head-mounted display device control method, an information system having a head-mounted display device, a head-mounted display device control method and a computer program for realizing the information system, and a record recording the computer program. It can be realized in the form of a medium and a data signal including a computer program thereof and embodied in a carrier wave.

It is explanatory drawing which shows schematic structure of the information system in 1st Embodiment of this invention. It is a block diagram which shows the structure of a server functionally. It is a list showing an example of advertising information matched with each of the pictures of the signboard memorized by advertising DB of a server. It is a block diagram which shows the structure of HMD functionally. It is a list showing an example of stay information of a user of HMD. It is explanatory drawing which shows a mode that image light is emitted by an image light generation part. It is a flow chart of advertisement display processing. It is explanatory drawing which shows an example of the captured image of a camera. It is explanatory drawing which shows an example of the visual field which a user visually recognizes, when the advertisement image of a detection signboard is displayed. It is explanatory drawing which shows an example of the visual field which a user visually recognizes after the visual line direction of the user of HMD changes. It is a block diagram which shows functionally the structure of HMD of 2nd Embodiment. It is a flow chart of advertising display processing in a 2nd embodiment. It is explanatory drawing which shows an example of a visual field which a user visually recognizes, when calibration setting is performed. It is explanatory drawing which shows an example of the visual field which a user visually recognizes, when the advertisement image of a detection signboard is displayed. It is explanatory drawing which shows an example of a visual field which a user visually recognizes after a calibration operation is performed. It is a block diagram which shows functionally the structure of HMD of 3rd Embodiment. It is a flow chart of advertising display processing in a 3rd embodiment. It is explanatory drawing which shows an example of the visual field which a user visually recognizes, when an advertising image is displayed in 3rd Embodiment. It is a flow chart of advertising display processing in a 4th embodiment. It is explanatory drawing which shows an example of a visual field which a user visually recognizes, when an advertising image is displayed in 4th Embodiment. It is a flow chart of advertising display processing of a 5th embodiment. It is explanatory drawing which shows an example of the visual field which a user visually recognizes, when an advertising image is displayed in 5th Embodiment. It is a flow chart of advertising display processing in a 6th embodiment. It is explanatory drawing which shows an example of the visual field which a user visually recognizes, when an advertising image is displayed in 6th Embodiment. It is a flow chart of advertising display processing in a modification. It is explanatory drawing which shows the external structure of HMD in a modification.

A. First embodiment:
A-1. Information system configuration:
FIG. 1 is an explanatory diagram showing a schematic configuration of an information system 500 according to the first embodiment of the present invention. The information system 500 is mounted on a user's head and allows a user to visually recognize a virtual image. A head mounted display 100 (Head Mounted Display, HMD100), an information terminal such as the HMD100, and signboards installed in various places, The server 300 stores the advertisement information including the position information for identifying each position of the signboard and the image information associated with each of the signboards. In the information system 500, the advertisement information stored in the server 300 is transmitted from the server 300 to the HMD 100 according to various information detected by the HMD 100. In the information system 500, only one HMD 100 and one server 300 are provided in the present embodiment, but in other embodiments, at least one of the HMD 100 and the server 300 may be provided in plurality.

FIG. 2 is a block diagram functionally showing the configuration of the server 300. The server 300 includes a storage unit 320 that stores advertisement information to be transmitted to the HMD 100, a communication unit 330 that transmits at least one copy of the advertisement information stored in the storage unit 320 to the HMD 100, and a CPU 310. I have it. The storage unit 320 includes a ROM, a RAM, and the like. The storage unit 320 has an image of a signboard and an advertisement database 325 (advertisement DB 325) that stores advertisement information associated with each of the images of the signboard. Details of the advertisement information stored in the advertisement DB 325 will be described later. The storage unit 320 also stores various computer programs.

The communication unit 330 performs wireless communication with other devices according to a predetermined wireless communication system such as a wireless LAN or Bluetooth (registered trademark). The communication unit 330 transmits and receives various information between the server 300 and the HMD 100. The CPU 310 reads out and executes the computer program stored in the storage unit 320. The CPU 310 is stored in the advertisement DB 325 based on the position information of the image display unit 20 of the HMD 100 received via the communication unit 330 and the information of the captured image of the camera 61 that captures the outside scene provided in the HMD 100. An information selection unit 312 that selects at least a part of the advertisement information is provided. A method of identifying the position of the image display unit 20 will be described later. The information selection unit 312 transmits selection information, which is a part of the selected advertisement information, to the HMD 100 via the communication unit 330. The server 300 in the present embodiment corresponds to the information storage device in the claims, and the communication unit 330 of the server 300 corresponds to the selection information transmission unit in the claims.

FIG. 3 is a list showing an example of advertisement information associated with each image of a signboard stored in the advertisement DB 325 of the server 300. In FIG. 3, a part of the advertisement data SD for each store of the advertisement information stored in the advertisement DB 325 is shown in a list. In the store-specific advertisement data SD, the category, the position of the signboard, and the image information are distinguished for each store name of the store. Although the details will be described later, the selection information transmitted from the server 300 to the HMD 100 is selected from the advertisement information based on the category and the position of the signboard. Also, the display mode of the image displayed on the HMD 100 is set based on the image information associated with the selection information transmitted from the server 300.

Hereinafter, the details of the information of the store-specific advertisement data SD will be described. For example, as shown in FIG. 3, the category of the store whose store name is "Fuji" is food and drink, and the category of the store whose store name is "Karaoke JO" is entertainment. The position of the signboard in the advertisement data SD for each store specifies the latitude and longitude of the place where the signboard is installed. For example, the latitude and longitude of the position of the signboard of the store whose store name is "Fuji" are "130.x...xx1" and "33.y...yy1", respectively.

The image information in the store-specific advertisement data SD is classified into the type of image, the number of pages, the time limit, and the environmental limit. The image type is the type of image displayed on the HMD 100. For example, the type of image of the store whose store name is “Fuji” is a “character” image (also simply referred to as “character image”). When displayed on the HMD 100, the character image is a still image represented by characters that does not change even if time elapses. In addition, the type of image of the store name “XX meat shop” is “moving image”, and the moving image is an image in which the image displayed on the HMD 100 changes with the passage of time. The number of pages indicates the hierarchical structure of images displayed on the HMD 100. For example, since the number of pages of the store whose store name is “Fuji” is “2”, it has a two-layer image structure. Therefore, although the image displayed on the HMD 100 is one type at a specific time, the image displayed on the HMD 100 is changed to another type image by receiving a predetermined operation. Since the number of pages in the store whose tempo information is "Karaoke JO" is "1", there is only one type of image displayed on the HMD 100.

The time limit is a setting for the time of the image displayed on the HMD 100. For example, since the time limit of the store whose store name is “Fuji” is from 11:00 to 20:00, the time when the image associated with “Fuji” is displayed on the HMD 100 is limited to from 11:00 to 20:00. At other times, the image associated with "Fuji" is not displayed. Further, the time limit of the store whose store name is "Karaoke JO" is two kinds from 10:00 to 18:00 and from 18:00 to 24:00. Therefore, the image associated with “Karaoke JO” displayed on the HMD 100 from 10:00 to 18:00 and the image associated with “Karaoke JO” displayed on the HMD 100 from 18:00 to 24:00 are different. Further, since there is no time limit for the store whose store name is “XX pharmacy”, the same image is displayed on the HMD 100 throughout the 24 hours of the day.

The environment restriction is the setting of the external environment of the HMD 100 regarding the condition for changing the image displayed on the HMD 100. For example, the environmental limitation of the store whose store name is “Fuji” is that the outside temperature is “30 degrees Celsius (30° C.)”, so the image displayed on the HMD 100 is different with 30° C. as the boundary. Further, since the environmental limitation of the store whose store name is "Karaoke JO" is "rain", the image displayed on the HMD 100 is different depending on whether the weather is rain or not. In addition, since the environmental limitation of the store whose store name is “XX pharmacy” is “none”, the image displayed on the HMD 100 is the same regardless of the external environment.

The HMD 100 shown in FIG. 1 is an optical transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly visually recognize an outside scene. In the present specification, the virtual image visually recognized by the user by the HMD 100 is also referred to as “display image” for convenience. In addition, emitting image light generated based on image data such as content is also referred to as “displaying an image”.

The HMD 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when mounted on the user's head, and a control unit 10 (controller 10) that controls the image display unit 20.

The image display unit 20 is a mounting body mounted on the head of the user, and has a spectacle shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. And a microphone 63 and a thermometer 65. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrows of the user when the user wears the image display unit 20.

The right holding portion 21 is provided by extending from the end portion ER which is the other end of the right optical image display portion 26 to a position corresponding to the user's temporal region when the user wears the image display portion 20. It is a member. Similarly, the left holding unit 23 extends from the end EL, which is the other end of the left optical image display unit 28, to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided by. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple (vine) of glasses.

The right display drive unit 22 and the left display drive unit 24 are arranged on the side facing the head of the user when the user wears the image display unit 20. Note that, hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as "optical image display unit".

The display drive units 22 and 24 include liquid crystal displays 241, 242 (Liquid Crystal Display, hereinafter also referred to as “LCDs 241 and 242”), projection optical systems 251, 252, and the like (see FIG. 4). The details of the configuration of the display drive units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 4) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display drive units 22 and 24 to the eyes of the user. The light control plate is a thin plate-shaped optical element, and is arranged so as to cover the front side of the image display unit 20, which is the side opposite to the eye side of the user. The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. Further, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the eyes of the user and adjust the visibility of the virtual image. The light control plate can be omitted.

The camera 61 is arranged at a position corresponding to the space between the eyebrows of the user when the user wears the image display unit 20. Therefore, the camera 61 captures the captured image by capturing an image of an outside scene that is an outside scene in the direction of the user's line of sight when the user wears the image display unit 20 on his/her head. The camera 61 of the present embodiment is set to an angle of view that is almost the same as the visual field of the user. The camera 61 is a monocular camera, but may be a stereo camera. The microphone 63 is a device that acquires external sound. The microphone 63 is arranged on the opposite side (outside) of the right display drive section 22 facing the user when the user wears the image display section 20. The thermometer 65 measures the temperature (Celsius) of the external environment. The thermometer 65 corresponds to the state detection unit in the claims.

The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connecting portion 40 includes a body cord 48 connected to the control portion 10, a right cord 42, a left cord 44, and a connecting member 46. The right cord 42 and the left cord 44 are cords in which the main body cord 48 is branched into two. The right cord 42 is inserted into the housing of the right holding unit 21 from the front end portion AP of the right holding unit 21 in the extending direction, and is connected to the right display drive unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the front end portion AP of the left holding unit 23 in the extending direction, and is connected to the left display driving unit 24. The connecting member 46 is provided at a branch point between the main body cord 48 and the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that fits with each other is provided at each of the end of the body cord 48 on the opposite side of the connecting member 46 and the control unit 10. By fitting/unmating the connector of the main body cord 48 and the connector of the control unit 10, the control unit 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the body cord 48, for example, a metal cable or an optical fiber can be adopted.

The control unit 10 is a device for controlling the HMD 100. The control unit 10 includes an enter key 11, a lighting unit 12, a display switching key 13, a track pad 14, a brightness switching key 15, a direction key 16, a menu key 17, and a power switch 18. There is. The enter key 11 detects a pressing operation and outputs a signal for determining the content operated by the control unit 10. The lighting unit 12 notifies the operating state of the HMD 100 by its light emitting state. The operating state of the HMD 100 includes, for example, turning the power on/off. As the lighting unit 12, for example, an LED is used. The display switching key 13 detects a pressing operation and outputs, for example, a signal for switching the display mode of the content moving image between 3D and 2D. The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal according to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The brightness switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the brightness of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up, down, left, and right directions and outputs a signal according to the detected content. The power switch 18 switches the power-on state of the HMD 100 by detecting the slide operation of the switch.

FIG. 4 is a block diagram functionally showing the configuration of the HMD 100. As illustrated in FIG. 4, the control unit 10 includes a storage unit 120, a power supply 130, a GPS module 137, a wireless communication unit 132, an operation unit 135, a CPU 140, an interface 180, and a transmission unit 51 (Tx51). And a transmitter 52 (Tx52).

The GPS module 137 receives the signal from the GPS satellite to identify the current position of the image display unit 20 and generate position information indicating the position of the image display unit 20. In other words, the GPS module 137 identifies the position of the HMD 100 including the image display unit 20. By specifying the current position of the image display unit 20, the current position of the user wearing the HMD 100 is specified. The GPS module 137 corresponds to the position specifying unit in the claims.

The wireless communication unit 132 performs wireless communication with other devices according to a predetermined wireless communication system such as a wireless LAN or Bluetooth (registered trademark). The wireless communication unit 132 transmits the position information of the image display unit 20 specified by the GPS module 137 to the communication unit 330 of the server 300. When the communication unit 330 receives the position information of the image display unit 20, the information selection unit 312 of the server 300 is stored in the advertisement DB 325 of the storage unit 320 and at a predetermined distance from the specified current position of the image display unit 20. The signboard image and advertisement information at the signboard position (FIG. 3) in is selected. The communication unit 330 uses the information of the signboard image and image information (FIG. 3) associated with the position of the signboard selected by the information selection unit 312 and the position of the signboard as selection information, and wireless communication of the HMD 100. To the section 132. The wireless communication unit 132 receives the selection information transmitted from the communication unit 330. In addition, the wireless communication unit 132 can acquire information about the weather condition and the current time of the specified current position of the image display unit 20 from a server different from the server 300. The wireless communication unit 132 corresponds to the information transmitting/receiving unit and the state detecting unit in the claims.

The power supply 130 supplies electric power to each unit of the HMD 100. As the power supply 130, for example, a secondary battery can be used.

The storage unit 120 stores various computer programs. The storage unit 120 is composed of a ROM, a RAM, and the like. The storage unit 120 has a selection information storage unit 122 and a past history storage unit 124. The selection information storage unit 122 stores the selection information transmitted from the server 300 via the wireless communication unit 132. An image associated with the image information included in the selection information is displayed on the image display unit 20 based on the selection information stored in the selection information storage unit 122, a captured image captured by the camera 61 described below, and the like.

The past history storage unit 124 stores past stay information of the HMD 100. The stay information is generated by the GPS module 137 specifying the current position of the HMD 100 in a predetermined cycle. Specifically, when the current position of the image display unit 20 that is cyclically specified is included within a predetermined range (for example, within 5 meters) from the position specified at a certain time point after a certain period of time. It is determined that the user of the image display unit 20 stayed at the specified position. Therefore, from the stay information stored in the past history storage unit 124, it is possible to determine the time when the user of the image display unit 20 stays at the store located at a specific position. The past history storage unit 124 corresponds to the personal information storage unit in the claims.

FIG. 5 is a list showing an example of the stay information VD of the user of the HMD 100. In the stay information VD, respective values are stored for items of a place of stay and a frequency of stay per month. In the item of stay location, the stay ID as a temporary name of the stay location and the latitude and longitude of the stay location are distinguished. Information on the stay frequency per month is created from the history of the most recent one month, which dates back to the day before. In the item of stay frequency, the total stay time per month, the number of stays per month, the average stay time of one stay, and the central average time that is the center average of the time when staying , Are distinguished. For example, as shown in FIG. 5, the latitude of the place A is “130.x...xx1” and the longitude of the place A is “33.y...yy1”. The user of the HMD 100 stays in the place A for a total of 240 minutes per month, and stays twice per month. Therefore, the average staying time per month is 120 minutes. Further, the central average time when the user stayed at the place A is 19:30. In the present embodiment, each of the place A, the place B, the place C, and the place D is a store of “Fuji”, “Karaoke JO”, “XX pharmacy”, and “XX meat store” in FIG. However, in another embodiment, it may be a different place.

The CPU 140 (FIG. 4) reads out and executes the computer program stored in the storage unit 120 to execute the operating system 150 (OS 150), the display control unit 190, the sound processing unit 170, the image processing unit 160, and the environmental information processing. It functions as the unit 169, the signboard detection unit 168, the past history collation unit 166, the distance measurement unit 161, and the display setting unit 165.

The display control unit 190 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 causes the right LCD control unit 211 to drive the right LCD 241 ON/OFF, the right backlight control unit 201 to drive the right backlight 221 ON/OFF, and the left LCD control unit according to control signals. The driving ON/OFF of the left LCD 242 by 212, the driving ON/OFF of the left backlight 222 by the left backlight control unit 202, and the like are individually controlled. Accordingly, the display control unit 190 controls the generation and emission of image light by the right display drive unit 22 and the left display drive unit 24, respectively. For example, the display control unit 190 causes both the right display drive unit 22 and the left display drive unit 24 to generate image light, only one of them to generate image light, and both of them not to generate image light.

The display control unit 190 transmits the control signals to the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. The display control unit 190 also transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

The image processing unit 160 acquires the image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. The image processing unit 160 also generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the cycle of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the sync signal has been separated into a digital image signal using an A/D conversion circuit or the like (not shown). After that, the image processing unit 160 stores the converted digital image signal in the DRAM in the storage unit 120 for each frame as image data (RGB data) of the target image. The image processing unit 160 may perform image processing such as resolution conversion processing, various color tone correction processing such as adjustment of brightness and saturation, and keystone correction processing on the image data as necessary. ..

The image processing unit 160 transmits the generated clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. .. The image data transmitted via the transmitter 51 is also referred to as “right-eye image data”, and the image data transmitted via the transmitter 52 is also referred to as “left-eye image data”. The transmission units 51 and 52 function as transceivers for serial transmission between the control unit 10 and the image display unit 20.

The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) in the left earphone 34 connected to the connecting member 46. (Not shown). Note that, for example, when the Dolby (registered trademark) system is adopted, the processing for the audio signal is performed, and the right earphone 32 and the left earphone 34 respectively output different sounds with different frequencies and the like. In addition, the voice processing unit 170 performs voice recognition on the external voice acquired by the microphone 63. When the external voice recognized by the voice recognition is the preset predetermined voice, the voice processing unit 170 transmits a signal indicating that the acquired voice is the predetermined voice to the operation unit 135. To do. The operation unit 135 executes various operations of the HMD 100 based on the signal transmitted from the voice processing unit 170 in addition to the key operation of pressing the enter key 11 or the like.

The signboard detection unit 168 uses the pattern matching or the statistical identification method to select an image of the signboard associated with the selection information stored in the selection information storage unit 122 from among the images captured by the camera 61 (hereinafter, simply (Also called “detection signboard”). In addition, the signboard detection unit 168 specifies the orientation of the camera 61 based on the position of the image display unit 20 specified by the GPS module 137 and the position of the detected signboard in the captured image. That is, the signboard detection unit 168 can specify the orientation of the image display unit 20 in which the camera 61 is arranged. When the signboard detection unit 168 detects an image of the signboard associated with at least one selection information from the captured images, the display setting unit 165 and the environmental information of the store-specific advertisement data SD of the store associated with the detected signboard are displayed. It is transmitted to the processing unit 169. The signboard detection unit 168, the camera 61, and the GPS module 137 correspond to the image detection unit in the claims, and the detected signboard image corresponds to the identification target image in the claims.

The environmental information processing unit 169 collates the temperature outside the HMD 100 measured by the thermometer 65 with the environmental restriction in the store-specific advertisement data SD transmitted from the signboard detection unit 168. The environmental information processing unit 169 transmits to the display setting unit 165 the matching result of the temperature outside the HMD 100 measured by the thermometer 65 and the environmental restriction in the store-specific advertisement data SD.

The distance measuring unit 161 uses the image display unit based on the current position of the image display unit 20 specified by the GPS module 137 and the position of the signboard stored in the selection information storage unit 122 and associated with the detected signboard. The distance between 20 and the position of the detected signboard is measured.

The past history collation unit 166 collates the store-specific advertisement data SD associated with the detected signboard with the stay information VD of the user of the HMD 100 stored in the past history storage unit 124. The past history collation unit 166 transmits the collation result between the store-specific advertisement data SD of the detected signboard and the stay information VD to the display setting unit 165.

The display setting unit 165 is displayed on the image display unit 20 based on the store-specific advertisement data SD associated with the detected signboard transmitted from the signboard detection unit 168 and the collation result transmitted from the environmental information processing unit 169. The display mode of the image to be displayed is set, and the image is displayed on the image display unit 20. A specific mode of the image displayed on the image display unit 20 will be described in "A-2. Advertisement display process" described later. The environmental information processing unit 169, the display setting unit 165, and the signboard detection unit 168 correspond to the display image control unit in the claims.

The operation unit 135 accepts a physical operation by a user, the enter key 11, the display switching key 13, the track pad 14, the brightness switching key 15, the direction key 16, the menu key 17, the power switch 18, and the voice processing unit. Various operations of the image display unit 20 are executed based on the signal which is voice-recognized and transmitted by the 170.

The interface 180 is an interface for connecting the control unit 10 to various external devices OA serving as a content supply source. The external device OA is, for example, a personal computer (PC), a mobile phone terminal, a game terminal, or the like. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. , A microphone 63, and a thermometer 65.

The right display drive unit 22 includes a reception unit 53 (Rx 53), a right backlight control unit 201 (right BL control unit 201) and a right backlight 221 (right BL 221) that function as a light source, and a right LCD functioning as a display element. It includes a control unit 211, a right LCD 241, and a right projection optical system 251. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as a display element.

The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is, for example, a light emitting body such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK input via the receiving unit 53, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data for the right eye. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 into parallel light fluxes. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path.

The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display drive unit 24 includes a reception unit 54 (Rx54), a left backlight control unit 202 (left BL control unit 202) and a left backlight 222 (left BL 222) that function as light sources, and a left LCD that functions as a display element. The controller 212 and the left LCD 242 and the left projection optical system 252 are included. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as a display element. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as an “image light generation unit”. Further, the left projection optical system 252 is configured by a collimator lens that converts the image light emitted from the left LCD 242 into a parallel light flux. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as a “light guide unit”.

FIG. 6 is an explanatory diagram showing how image light is emitted by the image light generator. The right LCD 241 changes the transmittance of light that passes through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, so that the illumination light IL emitted from the right backlight 221 is displayed as an image. Is modulated into an effective image light PL. The same applies to the left side. Note that, as shown in FIG. 6, although the backlight system is adopted in the present embodiment, the image light may be emitted by using the front light system or the reflection system.

A-2. Information detection process:
FIG. 7 is a flowchart of the advertisement display process. The advertisement display process is executed by the control unit 10 executing various processes. In the advertisement display process, the advertisement information associated with the detected signboard is displayed as an image on the image display unit 20 in a display mode set based on the temperature outside the HMD 100.

In the advertisement display process, first, when the operation unit 135 receives a predetermined operation, the application of the advertisement display process is activated (step S10). When the application of the advertisement display process is activated, the control unit 10 activates various devices such as the camera 61 necessary for executing the advertisement display process (step S12). Next, the activated GPS module 137 identifies the current position of the image display unit 20 (step S14). When the current position of the image display unit 20 is specified, the wireless communication unit 132 transmits the position information of the image display unit 20 to the communication unit 330 of the server 300 (step S16). Next, the wireless communication unit 132 receives the selection information selected by the server 300 based on the transmitted position information of the image display unit 20 (step S18).

When the wireless communication unit 132 receives the selection information, the camera 61 starts capturing an outside scene (step S20). Next, the signboard detection unit 168 monitors the detection of the same image as the image of the signboard included in the selection information received by the wireless communication unit 132 from the images captured by the camera 61 (step S22). In other words, the signboard detection unit 168 determines whether or not the detected signboard is included in the captured image. When the detected signboard is not included in the captured image (step S22: NO), the signboard detection unit 168 continuously monitors whether or not the detected signboard is included in the captured image (step S22). When the detected image includes the detected signboard (step S22: YES), the distance measurement unit 161 stores the current position of the image display unit 20 specified by the GPS module 137 and the selection information storage unit 122. Based on the position of the signboard, the distance between the image display unit 20 and the position of the detected signboard is measured (step S24).

FIG. 8 is an explanatory diagram showing an example of the captured image IMG of the camera 61. The captured image IMG shown in FIG. 8 is an image of the outside scene SC captured by the camera 61 arranged on the HMD 100 worn by the user. In the captured image IMG, the signboard detection unit 168 defines a signboard AB1, a signboard AB2, and a signboard AB3 as the same signboard as the signboard image included in the store-specific advertisement data SD of the selection information stored in the selection information storage unit 122. Detect 3 signs. As shown in FIG. 8, the "Fuji" signboard AB1 is installed in front of the entrance of the "Fuji" store, and the "Karaoke JO" signboard AB2 is installed above the "Karaoke JO" entrance. The signboard AB3 of "XX pharmacy" is installed in front of the entrance of the store of "XX pharmacy". Note that the signboard "XX Meat Shop" itself exists near the current position of the image display unit 20, but is not shown in the imaging range. The distance measurement unit 161 measures the distance between the current position of the image display unit 20 and the respective positions of the signboards AB1, AB2, AB3 detected by the signboard detection unit 168.

When the distance between the current position of the image display unit 20 and each position of the detected signboard is measured, the wireless communication unit 132, the thermometer 65, and the environmental information processing unit 169 acquire environmental information (see FIG. 7). Step S16). The environmental information includes the weather information and time of the current position of the image display unit 20 acquired by the wireless communication unit 132, and the temperature outside the image display unit 20 measured by the thermometer 65 and the environmental information processing unit 169. Is included. Next, the past history collation unit 166 collates the relationship between the stay information VD of the user of the HMD 100 stored in the past history storage unit 124 and the respective stores of the signboards AB1, AB2, AB3 as the detected signboards. (Step S28).

Next, the display setting unit 165 displays billboards AB1, AB2, AB3 that are detection billboards, advertisement data SD for each store of the billboards AB1, AB2, AB3, and the positions and image displays of the billboards AB1, AB2, AB3. A display mode for displaying the image of the advertisement included in the store-specific advertisement data SD on the image display unit 20 is set based on the distance from the unit 20, the environment information, and the stay information VD of the user ( Step S28). As an example of the set display mode, the display setting unit 165 enlarges the font of the character image associated with the detection signboard at a position close to the image display unit 20, or the current time and the user's An advertisement of a store whose center average time is close in the stay information VD is displayed in the central part of the range where the image light generation unit can generate image light, or an advertisement of a store where the stay time VD is long is conspicuously blinked. It is a mode. When the display mode is set, the display setting section 165 and the image processing section 160 cause the image light generation section to generate image light based on the set display mode and display the advertisement image (step S32). ..

FIG. 9 is an explanatory diagram showing an example of the visual field VR visually recognized by the user when the advertisement images TX1 and TX2 of the detected signboard are displayed. In FIG. 9, the advertisement image TX1 associated with “Fuji” and the “Karaoke JO” are associated with each other in the image display maximum area PN that is the maximum area in which the image light generation unit can generate image light. The advertising image TX2 and the advertising image TX2 that are viewed by the user of the HMD 100 are shown. The advertisement image TX1 is a character image showing today's member discount of “Fuji” and a recommended menu. The advertisement image TX2 is a character image indicating that the price of "Karaoke JO" per 30 minutes is 300 yen. Each of the advertisement images TX1 and TX2 is a character image associated with each of the signboards AB1 and AB2, and is displayed near the respective positions of the signboards AB1 and AB2 which are the detected signboards. Further, the respective distances from the current position of the image display unit 20 to the position of the signboard AB2 and the position of the signboard AB3 are substantially the same. On the other hand, the weather and the current time acquired by the wireless communication unit 132 are fine at 17:00. The external temperature acquired by the environmental information processing unit 169 and the thermometer 65 is 31°C. As shown in FIG. 3, since 30 degrees Celsius is set due to the environmental limitation of “Fuji” in the advertising data SD for each store, the maximum image display area PN (FIG. 9) corresponds to the case of 30° C. or higher. The advertisement image TX1 of "Fuji" is displayed. Further, since the current time is 17:00, it corresponds to any of the time restrictions of "Fuji", "Karaoke JO", and "XX pharmacy" in the store-specific advertisement data SD. However, in the “XX drugstore” in the stay information VD (FIG. 5), the average central time is 8:00, which is nine hours before the current time. In the present embodiment, the display setting unit 165 does not display the image of the advertisement in the image display maximum area PN when the average central time is 5 hours or more than the current time. Further, the numerical values of the staying time and the number of stays of the user per month in the stay information VD “Fuji” are larger than the staying time and the number of stays of the “Karaoke JO”. Therefore, the display setting unit 165 sets the font of the advertisement image TX1 associated with the signboard AB1 of “Fuji” to be thicker and larger than the font of the advertisement image TX2 associated with “Karaoke JO”. Since the number of pages in “Fuji” of the advertisement data SD for each store (FIG. 3) is set to “2”, the display setting unit 165 causes the image display to be performed by the operation unit 135 receiving a predetermined operation. The character image displayed on the advertisement image TX1 within the maximum area PN is changed. The advertising images TX1 and TX2 correspond to the images serving as the determination information corresponding information in the claims.

When the advertisement images TX1 and TX2 are displayed in the image display maximum area PN (step S32 in FIG. 7), the GPS module 137 detects a change in the current position of the image display unit 20 and the signboard detection unit 168 detects the camera 61. A change in the line-of-sight direction of the user of the HMD 100 based on the change in the captured image is detected (step S34). When a change is detected in at least one of the current position of the image display unit 20 and the line-of-sight direction of the user (step S34: YES), the control unit 10 repeats the processing from step S14.

FIG. 10: is explanatory drawing which shows an example of the visual field VR which a user visually recognizes after the visual line direction of the user of HMD100 changes. FIG. 10 shows a state in which the direction of the user's line of sight is further to the left than the state shown in FIG. The current position of the image display unit 20 is not changed from the state shown in FIG. As shown in FIG. 10, since the signboard AB2 of “Karaoke JO” is no longer the detected signboard, the display setting unit 165 sets the advertisement image TX2 associated with the signboard AB2 to non-display. Further, in this embodiment, since the image of the advertisement is set in advance so as not to be displayed in the central portion of the image display maximum area PN, the display setting unit 165 causes the advertisement of “Fuji” associated with the signboard AB1. The position to be displayed in the image display maximum area PN as the image TX3 is changed from the advertisement image TX1 (FIG. 9). In the state of FIG. 10, there is no change in time and weather as compared with the state of FIG. The advertisement image TX3 corresponds to the image as the determination information corresponding information in the claims.

In the processing of step S34 of FIG. 7, when it is determined that the current position of the image display unit 20 does not change and the user's line-of-sight direction does not change (step S34: NO), the control unit 10 Then, it is determined whether or not the advertisement display processing is ended (step S36). In the present embodiment, when the operation unit 135 receives a predetermined operation for ending the advertisement display process, the control unit 10 ends the advertisement display process. When the operation unit 135 does not receive the predetermined operation of ending the advertisement display process (step S36: NO), the control unit 10 repeats the process of step S34 and subsequent steps. In the process of step S36, when the operation unit 135 receives a predetermined operation for ending the advertisement display process (step S36: YES), the control unit 10 ends the advertisement display process.

As described above, in the HMD 100 of the present embodiment, the GPS module 137 specifies the current position of the HMD 100, and the wireless communication unit 132 transmits the specified position information of the HMD 100 to the server 300, and the server 300 transmits the position information. Receives the selected selection information. The signboard detection unit 168 and the GPS module 137 specify the orientation of the image display unit 20, and the display setting unit 165 displays the advertisement images TX1 and TX2 associated with the detected signboards AB1 and AB2 on the image display unit 20. Let Therefore, in the HMD 100 of the present embodiment, the user of the HMD 100 can visually recognize the advertisement images TX1 and TX2 associated with the detected signboard in addition to the actual signboards AB1 and AB2 in the visible range, and obtain more information. Therefore, the convenience for the user is improved. Further, the user can visually recognize the advertisement images TX1 and TX2 associated with the signboards AB1 and AB2 existing in the line-of-sight direction of the user, but even if the position of the signboard from the user is close, a direction deviating from the line-of-sight direction Since it is not necessary to visually recognize the information associated with the signboard existing in, it is possible to visually recognize only the necessary information.

In the HMD 100 of the present embodiment, the signboard detection unit 168 detects the signboards AB1 and AB2 from the image captured by the camera 61, and the display setting unit 165 is the advertisement associated with the signboards AB1 and AB2 of the detected signboard. The images TX1 and TX2 are displayed on the image display unit 20. Therefore, in the HMD 100 of the present embodiment, the advertisement image associated with the signboard that is not visually recognized by the user is not displayed on the image display unit 20 even if the HMD 100 is present in the direction of the user's line of sight, and the user does not need it. It is not necessary to see the advertisement image, and the convenience of the user is further improved.

In addition, in the HMD 100 of the present embodiment, the GPS module 137 acquires the information of the current time and the weather of the current position of the image display unit 20 from other servers as the environmental information, and the thermometer 65 and the environmental information processing unit 169 indicate The temperature outside the image display unit 20 is acquired, and the display setting unit 165 sets the advertisement images TX1 and TX2 to be displayed on the image display unit 20 according to the acquired environment information. Therefore, in the HMD 100 of the present embodiment, an image in which the information outside the image display unit 20 is also taken into consideration is displayed in the image display maximum area PN in addition to the current position of the image display unit 20 and the orientation of the image display unit 20, Since the user can visually recognize the information suitable for the situation on the spot, the convenience of the user is further improved.

Further, in the HMD 100 of the present embodiment, the past history storage unit 124 of the storage unit 120 stores the past stay information VD of the user, and the display setting unit 165 is detected in the stay information VD and the captured image IMG. The advertisement images TX1 and TX2 are displayed in the image display maximum area PN based on the store-specific advertisement data SD associated with the billboards AB1 and AB2. Therefore, in the HMD 100 of the present embodiment, in addition to the current position of the image display unit 20 and the orientation of the image display unit 20, the advertisement image based on the stay information VD of the user is displayed on the image display unit 20, so that the user Since the user can visually recognize information that suits his or her preference, the convenience of the user is further improved.

Further, in the HMD 100 of the present embodiment, the distance measuring unit 161 and the GPS module 137 measure the distance between the current position of the image display unit 20 and the position of the detected signboard, and the display setting unit 165 determines the distance based on the measured distance. The font sizes of the character images of the advertisement images TX1 and TX2 associated with the signboards AB1 and AB2 which are the detected signboards are set. Therefore, in the HMD 100 according to the present embodiment, the larger the character image included in the advertisement information associated with the signboard that is closer to the image display unit 20, the larger the image is displayed on the image display unit 20, and the user is displayed on the image display unit 20. The advertisement images TX1 and TX2 can recognize the distance from the position of the detected signboard, further improving the convenience for the user.

B. Second embodiment:
B-1. HMD configuration:
FIG. 11 is a block diagram functionally showing the configuration of the HMD 100a of the second embodiment. The HMD 100a of the second embodiment is different from the HMD 100 of the first embodiment in that the storage unit 120a of the control unit 10a does not have the past history storage unit 124 and that the CPU 140a of the control unit 10a has a past history collation unit. 166 and the environment information processing unit 169, instead of functioning as a calibration setting unit 167, and the image display unit 20a does not include a thermometer 65. Other configurations are the same as the HMD 100 of the first embodiment. Is the same as.

The calibration setting unit 167 detects the index finger of the user of the HMD 100a as a specific object detected in the captured image of the camera 61, and the advertisement image displayed in the image display maximum area PN of the image display unit 20a. , Define a relative positional relationship. A specific definition method will be described in “B-2. Advertisement display process”. The calibration setting unit 167 corresponds to the calibration unit in the claims.

B-2. Advertising display processing:
FIG. 12 is a flowchart of the advertisement display process in the second embodiment. In the advertisement display processing of the second embodiment, compared with the advertisement display processing of the first embodiment (FIG. 7), the processing of step S44 and step S62 is added, and step S26 and step S28 of FIG. The other processing is the same as the advertisement display processing of the first embodiment, except that the processing is deleted. Therefore, in the second embodiment, detailed description of the same processing as the advertisement display processing of the first embodiment will be omitted.

In the advertisement display process of the second embodiment, when the application of the advertisement display process is activated (step S40 in FIG. 12), after the CPU 140a activates various devices (step S42), the calibration setting unit 167 of the CPU 140a sets the calibration. Is executed (step S44). In the calibration setting, first, the display setting unit 165 displays a setting image, which is an image for executing the calibration setting, in the image display maximum area PN of the image display unit 20a. After that, the camera 61 starts capturing an image. The calibration setting unit 167 changes the position of the index finger of the user preset as a specific object detected in the captured image of the camera 61 and the setting image displayed in the image display maximum area PN. Perform calibration settings based on the relationship.

FIG. 13 is an explanatory diagram showing an example of the visual field VR visually recognized by the user when the calibration setting is performed. FIG. 13 shows a state in which the setting image CG for performing the calibration setting is displayed in an overlapping manner on the outside scene SC, and the user's right hand HD is captured in the captured image of the camera 61. The setting image CG includes a rectangular image BG, five circular images P1, P2, P3, P4 and P5, and four arrow images CS1, Cs2, CS3 and CS4. The circular image P1 is formed near the upper left corner of one corner of the square image BG, the circular image P2 is formed near the upper right corner of the square image BG, and the circular image P3 is formed at the lower right corner of the square image BG. The circular image P4 is formed in the vicinity of the apex, the circular image P4 is formed in the vicinity of the lower left apex of the square image BG, and the circular image P5 is formed in the center of the square image BG. The arrow image CS1 is a straight line arrow from the circular image P1 to the circular image P2, the arrow image CS2 is a straight line arrow from the circular image P2 to the circular image P3, and the arrow image CS3 is a circular shape. The arrow from the image P3 to the circular image P4 is a straight line, and the arrow image CS4 is from the circular image P4 to the circular image P5.

The calibration setting unit 167 detects the index finger FF of the right hand HD in the captured image after the setting image CG is displayed in the image display maximum area PN. After that, the calibration setting unit 167 sequentially sets the circular image P1, the circular image P2, and the circular image P2 along the arrow image CS1, the arrow image CS2, the arrow image CS3, and the arrow image CS4 with the index finger FF as the starting point. By detecting a motion that overlaps the image P3, the circular image P4, and the circular image P5, the positional relationship between the setting image CG and the index finger FF included in the outside scene SC is defined. Thereby, the calibration setting unit 167 sets which pixel in the image display maximum area PN the index finger FF imaged by the camera 61 corresponds to. The set image CG corresponds to the specific image in the claims, and the index finger FF of the right hand HD corresponds to the specific object in the claims.

When the calibration setting is executed (step S44 in FIG. 12), the control unit 10a performs the processing from step S46 to step S56. After that, the display setting unit 165 displays an image based on the detected signboard, the advertisement data SD for each store of the detected signboard, and the distance between each position of the detected signboard and the current position of the image display unit 20. The display mode of the advertisement image displayed on the unit 20 is set (step S58). Next, the display setting unit 165 and the image processing unit 160 display the advertisement image on the image display unit 20 based on the set display mode (step S58).

FIG. 14 is an explanatory diagram showing an example of the visual field VR visually recognized by the user when the advertisement images TX2 and TX4 of the detected signboard are displayed. As shown in FIG. 14, in the second embodiment, the display setting unit 165 displays the advertisement image TX4 associated with the signboard AB1 which is the detected signboard in the image display maximum area PN. The advertisement image TX4 includes two selectable icon images IC1 and IC2. After the calibration setting, the user performs a calibration operation of superimposing the index finger FF on the icon image IC1 or the icon image IC2, whereby the operation unit 135 selects the superimposed icon image and selects the selected icon image. An advertisement image different from the advertisement image TX4 associated with can be displayed in the image display maximum area PN.

When the advertisement image is displayed in the image display maximum area PN (step S60 in FIG. 12), the operation unit 135 determines whether or not a voice input or a calibration operation by the index finger FF has been accepted (step S62). Note that in other embodiments, it may be determined whether a key operation such as the enter key 11 of the operation unit 135 has been accepted. In the process of step S62, when the operation unit 135 receives the voice input or the calibration operation (step S62: YES), the display setting unit 165 determines the maximum image display based on the received voice input or the calibration operation. The display mode of the advertisement image displayed in the region PN is set (step S58), and the display setting unit 165 and the image processing unit 160 display the advertisement image set in the image display maximum region PN (step S60).

FIG. 15 is an explanatory diagram showing an example of the visual field VR visually recognized by the user after the calibration operation is performed. FIG. 15 shows an advertisement image TX5 displayed when the icon image IC2 in the advertisement image TX4 shown in FIG. 14 is selected by the calibration operation. The advertisement image TX5 shows a part of the menu of the drink provided in “Fuji” associated with the selected icon image IC2 (FIG. 14) by a character image, and includes two icon images IC1 and IC3. I'm out. When the icon image IC3 is selected by voice input or calibration operation, the display setting unit 165 displays the advertisement image TX4 in the image display maximum area PN instead of the advertisement image TX5. Further, when the icon image IC1 is selected, the display setting unit 165 displays a character image of a part of the menu of the dish provided by “Fuji” associated with the icon image IC1 instead of the advertisement image TX5. It is displayed in the maximum display area PN. The advertisement images TX4 and TX5 correspond to the images serving as the determination information corresponding information in the claims.

When the advertisement images TX2 and TX5 are displayed in the image display maximum area PN (step S60 in FIG. 12), the operation unit 135 determines whether or not a voice input or a calibration operation by the index finger FF is accepted (step S62). .. When the operation unit 135 does not receive the voice input or the calibration operation (step S62: NO), the control unit 10a performs the processing of step S64 and subsequent steps.

As described above, in the HMD 100a of the second embodiment, the calibration setting unit 167 includes the square image BG displayed on the image display unit 20a and the index finger FF of the user's right hand HD detected in the captured image. The calibration setting is performed based on the positional relationship with. After the calibration setting, the display setting unit 165 changes the image displayed on the image display unit 20a from the advertisement image TX4 to the advertisement image TX5 based on the position of the index finger FF detected in the captured image. Therefore, in the HMD 100a of the second embodiment, by superimposing the actual outside scene SC and the image displayed on the image display unit 20a, the user recognizes the outside scene SC and the displayed image as the same and operates them. This improves the usability for the user.

C. Third embodiment:
C-1. HMD configuration:
FIG. 16 is a block diagram functionally showing the configuration of the HMD 100b of the third embodiment. In the first and second embodiments, the advertisement image is displayed in the image display maximum area PN when the detected signboard is included in the captured image. However, in the third embodiment, the current image of the identified user is displayed. The advertisement image of the image display maximum area PN is displayed based on the position and the determination information for determining the presence or absence of the image display. Therefore, in the third embodiment, a configuration different from the configurations of the first and second embodiments will be described, and description of the same configuration will be omitted. In the third embodiment, the past history of the user is used as the judgment information.

As shown in FIG. 16, the image display unit 20b includes a right-eye imaging camera 37 that captures the right eye of a user who wears the image display unit 20b on his head, and a left-eye imaging camera that captures the left eye of the user. 38, and the biometric information acquisition part 39 which acquires some biometric information of a user. In the third embodiment, the right-eye imaging camera 37 and the left-eye imaging camera 38 are monocular cameras, but they may be stereo cameras. Hereinafter, the right-eye imaging camera 37 and the left-eye imaging camera 38 are collectively referred to simply as “eye-imaging cameras 37, 38”. The images of the user's eyes captured by the eye imaging cameras 37 and 38 are used for specifying the direction of the user's line of sight and measuring the size of the user's pupil, which will be described later. The biometric information acquisition unit 39 is a sensor that acquires at least one of biometric information of the user such as body temperature, blood pressure, heart rate, respiration, and electroencephalogram state of the user. Details of the biological information acquisition unit 39 and the environmental information processing unit 169b in the CPU 140b of the control unit 10b that processes biological information will be described in a fourth embodiment described later.

Like the first embodiment, the past history storage unit 124b of the storage unit 120b of the third embodiment, in addition to the stay information in the past of the HMD 100b, includes the product purchased by the user at a specific store and the price of the product. It also stores purchasing information. The purchase information is input by the user performing a predetermined operation on the operation unit 135, or the information of the credit card used in a specific store is acquired via the wireless communication unit 132, so that the past It is stored in the history storage unit 124b.

The CPU 140b of the control unit 10b includes a determination unit 162 that determines the advertisement image to be displayed in the image display maximum area PN based on the acquired captured image of the eye and the acquired biometric information. The determination unit 162 communicates with the server 300 via the wireless communication unit 132 to determine whether or not the position of the HMD 100b specified by the GPS module 137 is within the specific store in the advertisement DB 325. When the determination unit 162 determines that the position of the HMD 100b is inside a specific store, the determination unit 162 acquires display information as selection information regarding the display of the advertisement image or the product of the specific store via the wireless communication unit 132. To do. In this case, the past history collating unit 166b collates the stay information and the purchase information stored in the past history storage unit 124b to collate the past history of the user at a specific store, which is the current position of the user. To do.

The determination unit 162 specifies the line-of-sight direction of the user with respect to the captured image of the user's eye by pattern matching or a statistical identification method. In addition, the determination unit 162 specifies the change in the size of the pupil by measuring the size of the pupil. The determination unit 162 estimates the predicted behavior of the user or the intention of the user based on the collation result of the past history collation unit 166b and the identified line-of-sight direction and pupil size of the user. The determination unit 162 displays the advertisement image determined based on the estimated behavior of the user or the intention of the user in the image display maximum area PN. Specifically, for example, the determination unit 162 estimates that the user intends to purchase a product in the user's past purchase history and a product near the specified line-of-sight direction, An advertisement image associated with the product is selected. In addition, the determination unit 162 may additionally display an advertisement image related to a special sale of a specific store, which is the current position of the user, in the image display maximum area PN. Similar to the display setting unit 165 of the first embodiment, the display setting unit 165b sets the display mode of the advertisement image and other images to be displayed in the image display maximum region PN and displays the image in the image display maximum region PN. Let The captured image of the user's eye and the past purchase history of the user in the third embodiment correspond to the determination information in the claims. The determination unit 162 corresponds to the determination information acquisition unit in the claims.

C-2. Advertising display processing:
FIG. 17 is a flowchart of the advertisement display process according to the third embodiment. In the advertisement display process of the third embodiment, compared with the advertisement display process of the first embodiment, the processes from step S70 to step S84 are different, and the other processes are the same. Therefore, in the third embodiment, the processing from step S70 to step S84 different from the first embodiment will be described, and the description of other processing will be omitted.

In the process of step S16 of FIG. 17, when the wireless communication unit 132 transmits the position information of the image display unit 20b, which is the position information of the user, to the server 300, the wireless communication unit 132 detects that the position information of the image display unit 20b near the position information of the image display unit 20b is displayed from the server 300. The location information of the store is received (step S70). The determination unit 162 acquires the position information of the store associated with the current position of the image display unit 20b via the wireless communication unit 132. The determination unit 162 determines whether or not the current position of the image display unit 20b specified as the current position of the user is within the specific store (step S72). When the determination unit 162 determines that the current position of the image display unit 20b is not within the specific store (step S72: NO), the determination unit 162 performs the same processes from step S34 onward as in the first embodiment.

In the process of step S72, when the determination unit 162 determines that the current position of the image display unit 20b is within the specific store (step S72: YES), the images captured by the eye imaging cameras 37 and 38 are captured. Based on this, the line-of-sight direction of the user is specified (step S73). The determination unit 162 receives, from the server 300 via the wireless communication unit 132, the display information as the selection information associated with the specified user's line-of-sight direction and the specific store at the current position (step). S74). Next, the past history collation unit 166b collates the purchase information of the user stored in the past history storage unit 124b. The determination unit 162 determines whether or not there is a purchase history of previous purchase of a product displayed near the identified user's line-of-sight direction, based on the comparison result of the past history comparison unit 166b (step). S78). When the determination unit 162 determines that there is a purchase history of the product near the line-of-sight direction (step S78: YES), the determination unit 162 estimates that the user has the intention to purchase the product and associates it with the purchase history. The selected image is determined as the display image, and the display mode of the image is set (step S80). After that, the determination unit 162 determines the advertisement image associated with the store including the current position of the user and the purchase history, and sets the display mode of the determined advertisement image (step S84). In addition, in the process of step S78, when the determination unit 162 determines that there is no purchase history of the product near the line-of-sight direction (step S78: NO), the advertisement image of the store including the current position of the user is determined. Then, the display mode of the determined advertisement image is set (step S82). After that, the display setting unit 165b causes the image including the advertisement image to be displayed in the image display maximum area PN based on the set display mode (step S84), and repeats the processing from step S34.

FIG. 18 is an explanatory diagram showing an example of the visual field VR visually recognized by the user when the advertisement images TX6 and TX7 are displayed in the third embodiment. In FIG. 18, when the user's current position is a shopping center as a specific store and the user's line-of-sight direction is toward the “flower” sales floor in the shopping center, the user visually recognizes it as a visual field VR. The outside scene SC and the advertisement images TX6 and TX7 are shown. The advertisement image TX6 is a character image associated with the purchase history and displayed in the image display maximum area PN. The advertisement image TX6 has a history that the user purchased a flower stored in the advertisement DB 325 as a “seasonal flower” one week ago, and is registered in advance as a “seasonal flower” in the current shopping center. The names of the three types of flowers are "tulip", "morning glory", and "rose". Further, the advertisement image TX7 is an image that is stored in the advertisement DB 325 and represents the advertisement information associated with the “flower” sales floor of the shopping center in the image display maximum area PN with characters. Advertisement image TX7 advertises two types of flowers, "yuri" and "chrysanthemum" as today's special sale items at the shopping center.

In the third embodiment, the display setting unit 165b sets the advertisement image TX6 associated with the purchase history related to the personal information of the user as the position where the advertisement image is displayed in the maximum image display area PN. It is displayed near the center of the region PN. Further, the display setting unit 165b causes the advertisement image TX6, which is a part of the advertisement information transmitted from the advertisement DB 325, to be displayed at a position other than the center of the maximum image display area PN due to the current position of the user. In other embodiments, the position displayed in the image display maximum area PN can be variously modified depending on the type of advertisement image. The advertisement images TX6 and TX7 in the third embodiment correspond to the images as the judgment information corresponding information in the claims.

As described above, in the HMD 100b according to the third embodiment, the determination unit 162 images the advertisement image TX6 shown in FIG. 18 based on the collation result of the past history collation unit 166b and the identified line-of-sight of the user. It is displayed in the maximum display area PN. In the HMD 100b of the third embodiment, since the purchase history, which is one of the personal information of the user, is referred to when the advertisement image displayed in the image display maximum area PN is determined, the relationship with the user is high. It is not necessary for the user to see the low advertisement image, and the convenience of the user is improved.

D. Fourth Embodiment:
In the fourth embodiment, the HMD 100b uses the biometric information of the user acquired by the biometric information acquisition unit 39 of the image display unit 20b illustrated in FIG. 16 and the selection information received from the server 300 to determine the maximum image display area. Display the advertisement image on the PN. In the fourth embodiment, the biometric information acquisition unit 39 is a sweat detection sensor that detects the sweat of the user by being attached to the skin of the user. Humans sweat when a chemical reaction that produces lactic acid occurs in the body due to muscle activity. The biometric information acquisition unit 39 measures the generated lactic acid value from the skin of the user, so that the user detects the amount of sweat.

FIG. 19 is a flowchart of the advertisement display process in the fourth embodiment. The advertisement display process of the fourth embodiment includes the processes of steps S90 and S92 instead of the processes of steps S70 to S80, as compared with the advertisement display process of the third embodiment shown in FIG. The difference is that the process of step S34 is not provided, and other processes are the same as the process of the advertisement display process of the third embodiment. Therefore, in the advertisement display processing of the fourth embodiment, the processing from step S90 to step S84 different from the advertisement display processing of the third embodiment will be described, and the description of other processing will be omitted.

In the process of step S16 of FIG. 19, when the wireless communication unit 132 transmits the position information of the image display unit 20b, which is the position information of the user, to the server 300, the biometric information acquisition unit 39 determines the biometric information of the user. , The sweat of the user is detected (step S90). Next, the determination unit 162 determines whether or not the detected sweat of the user is equal to or more than the amount of sweat as a preset threshold value (step S92). When the determination unit 162 determines that the detected amount of sweat of the user is not equal to or greater than the threshold value (step S92: NO), the same processing from step S36 onward as in the first and third embodiments is repeated.

In the process of step S92, when the determination unit 162 determines that the detected amount of sweat of the user is equal to or larger than the threshold value (step S92: YES), the determination unit 162 corresponds to the amount of sweat of the user equal to or larger than the threshold value in advance. The attached selection information is received from the server 300 (step S74). In the fourth embodiment, when the determination unit 162 detects the amount of sweat that is equal to or more than the threshold value of the user, the determination unit 162 estimates that the user is thirsty, and determines that the coffee shop centered on the current position of the user. The display mode of the image showing the position, the advertisement image of the coffee shop, and the position of the drink vending machine is set (step S82). After that, the display setting unit 165b causes the image including the advertisement image to be displayed in the image display maximum area PN based on the set display mode (step S84), and repeats the processing from step S36.

FIG. 20 is an explanatory diagram showing an example of the visual field VR visually recognized by the user when the advertisement image GM is displayed in the fourth embodiment. FIG. 20 shows the outside scene SC included in the visual field VR of the walking user and the advertisement image GM displayed in the image display maximum area PN. The advertisement image GM includes a drink vending machine (“Automato” in FIG. 20) and a coffee shop (see FIG. 20) displayed in the image display maximum area PN when the amount of sweat exceeding the threshold of the user is detected. 21 is a schematic image of a bird's-eye view showing the positional relationship between the user (“Shop”) and the user (“User” in FIG. 20). The determination unit 162 determines, based on the position information of the user and the selection information received from the server 300, the position of the vending machine and the position of the coffee shop existing within a predetermined range from the position of the user. The advertisement image GM compared with the position is displayed in the image display maximum area PN. The advertising image GM in the fourth embodiment corresponds to the image as the judgment information corresponding information in the claims.

As described above, in the HMD 100b of the fourth embodiment, the determination unit 162 sets the advertisement image GM to be displayed in the image display maximum area PN based on the amount of sweat of the user as the biometric information of the user. .. Therefore, in the HMD 100b of the fourth embodiment, the user can visually recognize an image according to the physical condition or the like of the user at the present time, and the convenience of the user is improved.

E: Fifth embodiment:
In the fifth embodiment, as compared with the third embodiment, the determination unit 162 determines that the maximum image display area PN based on the statistical information of a plurality of other users instead of the purchase history which is the user's personal information. The setting of the advertisement image to be displayed is different, and the other processes in the configuration and the advertisement display process are the same as those in the third embodiment.

In the fifth embodiment, the advertisement DB 325 of the server 300 has statistical information of other users who are different users from the plurality of users. The statistical information includes, for example, the number of people entering the amusement park every hour, the waiting time for playground equipment such as a roller coaster, and the like.

FIG. 21 is a flowchart of the advertisement display process of the fifth embodiment. In the advertisement display process of the fifth embodiment, compared with the advertisement display process of the third embodiment shown in FIG. 17, the processes from step S100 to step S106 are different, and other processes are the same as those of the third embodiment. Is. Specifically, in the advertisement display process of the fifth embodiment, compared to the advertisement display process of the third embodiment shown in FIG. 17, the current position of the user is not in a shopping center as a specific store, but in a tourist spot. In front of the information bulletin board and that the advertisement image displayed in the maximum image display area PN is set based on statistical information different from the purchase information. Therefore, in the fifth embodiment, the processing from step S100 to step S84 including processing different from the advertisement display processing of the third embodiment will be described, and description of other processing will be omitted.

In step S16 of FIG. 21, when the wireless communication unit 132 transmits the position information of the image display unit 20b, which is the position information of the user, to the server 300, the store or signboard near the position of the image display unit 20b from the server 300. Position information of various facilities including is received (step S100). The determination unit 162 identifies the positional relationship with the facility closest to the current position of the user via the wireless communication unit 132 (step S102). When the determination unit 162 identifies that the current position of the user is in front of the information bulletin board of the tourist destination which is one of various facilities (step S102: YES), the user of the eye imaging cameras 37, 38 The eye gaze direction and the size of the pupil of the user are specified using the imaged image of the eye (step S103). The determination unit 162 receives the selection information associated with the information board of the tourist attraction as the detected signboard from the server 300 via the wireless communication unit 132 (step S104). The determination unit 162 sets the display mode of the advertisement image to be displayed in the image display maximum area PN based on the selection information received from the server 300 and the identified line-of-sight direction of the user (step S82). Then, the display setting unit 165b displays the advertisement image in the image display maximum area PN based on the set display mode (step S84).

FIG. 22 is an explanatory diagram showing an example of the visual field VR visually recognized by the user when the advertisement image TX8 is displayed in the fifth embodiment. FIG. 22 shows the outside scene SC included in the visual field VR of the user, and the guide image CI and the advertisement image TX8 displayed in the image display maximum area PN. In addition, the outside scene SC includes the guidance bulletin board GU that the user actually visually recognizes. In the fifth embodiment, the guide board CI as the advertisement image and the advertisement image TX8 are associated with the guide bulletin board GU which is the detected signboard. The guide image CI is an image representing a concierge for guiding a tourist destination shown on the guide bulletin board GU. Further, the advertisement image TX8 is a character image representing a recommended tourist spot based on the statistical information created by the history of other users stored in the advertisement DB 325. The statistical information in the fifth embodiment corresponds to the judgment information in the claims, and the guide image CI and the advertisement image TX8 correspond to the images as the judgment information correspondence information in the claims.

In the process of step S84 of FIG. 21, when the determination unit 162 displays the guide image CI and the advertisement image TX8 in the image display maximum area PN, the size of the user's pupil is determined based on the captured image of the user's eye. Is determined (step S106). When the determination unit 162 determines that the change in the size of the user's pupil does not detect a change in which the pupil becomes larger than the threshold value (step S106: NO), the same process as in step S34 and subsequent steps in the third embodiment is performed. Repeat the process.

In the process of step S106, when the determination unit 162 determines that the change in which the size of the user's pupil becomes larger than the threshold is detected (step S106: YES), the determination unit 162 in the fifth embodiment. Presumes that the user was interested in the image or the actual object in the line-of-sight direction of the user, and resets the display mode of the advertisement image displayed in the image display maximum area PN (step S82). .. Specifically, the determination unit 162 displays a larger image associated with an object (for example, “□□ river” in FIG. 22) in the user's line-of-sight direction, or is associated with the object. More detailed information is displayed in the image display maximum area PN.

As described above, in the HMD 100b of the fifth embodiment, the determination unit 162 sets the advertisement image to be displayed in the image display maximum area PN based on the statistical information based on a plurality of other persons different from the user. .. Therefore, in the HMD 100b of the fifth embodiment, the user can visually recognize the advertisement image using the statistical information of other humans at a tourist spot where the user has never visited or a place where the user has little information. This improves the convenience of the user.

Further, in the HMD 100b of the fifth embodiment, the determination unit 162 estimates the degree of interest of the user according to the change in the size of the user's pupil, and determines the advertisement image to be displayed in the image display maximum area PN. Since the setting is made, the convenience for the user is further improved.

F. Sixth embodiment:
The sixth embodiment is different from the third embodiment in that the determination unit 162 sets the advertisement image to be displayed in the image display maximum area PN based on the external sound acquired by the microphone 63, and is different from the third embodiment. The configuration and processing of is the same as that of the third embodiment. In the sixth embodiment, when the user is talking to a friend who is another person, the determination unit 162 uses the words that frequently appear in the conversation, the time when the conversation is performed, and the like. The advertisement image related to the content that the person and the friend guess is desired is displayed in the image display maximum area PN. For example, the determination unit 162, in the evening, in a conversation between the user and a friend, when a “meal” and a word highly related to the “meal” are frequently acquired, a nearby restaurant that can perform the “meal”. An advertisement image such as a bar or a bar is displayed in the image display maximum area PN.

FIG. 23 is a flowchart of the advertisement display process according to the sixth embodiment. The advertisement display process of the sixth embodiment is different from the advertisement display process of the third embodiment in the processes from step S110 to step S114, and the other processes are the same as those in the third embodiment. Therefore, in the advertisement display processing of the sixth embodiment, the processing from step S110 to step S84 including processing different from the advertisement display processing of the third embodiment will be described, and description of other processing will be omitted.

When the process of step S16 of FIG. 23 is performed, microphone 63 acquires an external voice (step S110). The determination unit 162 identifies a preset word from the acquired external voice and extracts the word (step S112). The determination unit 162 receives, from the server 300, selection information that is highly related to the extracted words, via the wireless communication unit 132, according to the number of extracted words (step S114). The determination unit 162 sets the display mode of the advertisement information included in the received selection information to be displayed in the image display maximum area PN according to the frequency of the extracted words (step S82). The display setting unit 165b displays the advertisement image in the image display maximum area PN based on the set display mode (step S84).

FIG. 24 is an explanatory diagram showing an example of the visual field VR visually recognized by the user when the advertisement image TX9 is displayed in the sixth embodiment. In FIG. 24, the advertisement image TX9 and the outside scene SC displayed in the image display maximum area PN when the words extracted from the voice acquired by the microphone 63 are “meal”, “fish”, and “Japanese food”. The visual field VR of the user including and is shown. In the sixth embodiment, the determination unit 162, based on the words extracted from the conversation acquired from the conversation between the user and the friend FR, as the advertisement image TX9, a character image showing a list of shops where dining is possible. The image is displayed in the maximum image display area PN. In the advertisement image TX9, a Japanese restaurant with a store name of “Fuji”, a Japanese restaurant with a store name of “Shunyo”, a Chinese restaurant with a store name of “Cantonese”, and a bar with a store name of “Hide” And are shown as a list of stores. The acquired voice in the sixth embodiment corresponds to the determination information in the claims, and the advertisement image TX9 corresponds to the image as the determination information corresponding information in the claims.

As described above, in the HMD 100b of the sixth embodiment, the determination unit 162 sets the advertisement image to be displayed in the image display maximum area PN based on the word extracted from the acquired voice. Therefore, in the HMD 100b according to the sixth embodiment, the information that the user is supposed to want is automatically displayed as the advertisement image TX9 in the image display maximum area PN based on the voice, which is convenient for the user. Further improve.

G. Modification:
The present invention is not limited to the above embodiment, and can be implemented in various modes without departing from the scope of the invention, and the following modifications are possible, for example.

G-1. Modification 1:
In the above embodiment, the orientation of the image display unit 20 is specified based on the position of the image display unit 20 specified by the GPS module 137 and the image of the signboard detected in the captured image of the camera 61. The method for specifying the orientation of the image display unit 20 can be modified in various ways. For example, the image display unit 20 is equipped with an orientation sensor and an acceleration sensor, and based on the current position of the image display unit 20 and the orientation of north, south, east, and west specified by the orientation sensor, and the direction with respect to the gravity direction specified by the acceleration sensor. The orientation of the image display unit 20 may be specified. In the HMD 100 of this modification, even if there is an obstacle (for example, a car) that temporarily blocks the front of the user, an image associated with the signboard detected when there is no obstacle is displayed on the image display unit 20. Can be displayed on. Further, only the image associated with the signboard that is not visually recognized by the user is displayed on the image display unit 20, so that the user is associated with the signboard visually recognized as the real thing and the signboard not visually recognized as the real thing. Images can be recognized and a wide range of information can be acquired.

Further, in the above embodiment, an example of the advertising data SD for each store (FIG. 3) and the stay information VD (FIG. 5) is shown, but the items in the advertising data SD for each store and the stay information VD are the items of the above embodiment. It is not limited to the above, and various modifications are possible. For example, there may be restrictions based on humidity and age as an example of the item of the advertisement data SD for each store. Further, the past history storage unit 124 of the storage unit 120, instead of the stay information VD of the user, the frequency of the advertisement image displayed on the image display unit 20 in the past of the user, the number of times the store has been used, and the wireless communication unit 132. You may memorize the number of times the store was searched using. Further, the storage unit 120 stores personal information such as sex, hobbies, addresses, and occupations, not the past information of the user, and the display setting unit 165 stores the image display unit 20 based on the stored personal information. You may set the advertisement image displayed on.

Further, in the above-described embodiment, the distance from the image display unit 20 to the position of the detected signboard is measured based on the current position of the image display unit 20 and the position of the detected signboard acquired by the GPS module 137. The method of measuring the distance between the current position of the display unit 20 and the position of the detected signboard can be variously modified. For example, as the distance measuring unit 161, an infrared light emitting unit and an infrared light receiving unit are arranged near the camera 61 of the image display unit 20, and the infrared light receiving unit receives the infrared light emitted by the infrared light emitting unit. May measure the distance between the image display unit 20 and the object that reflects infrared rays.

Further, in the above embodiment, the operation unit 135 executes various operations of the HMD 100 based on the voice signal recognized by the voice processing unit 170, but controls other than the operation are performed based on the recognized voice signal. May be performed. For example, the display setting unit 165 may set the advertisement image to be displayed on the image display unit 20, based on the signboard detected in the captured image and the recognized voice. Specifically, when the voice of "hungry" is recognized, the display setting unit 165 sets the advertisement information associated with the detected signboard having only the category "food and drink" in the advertisement data SD for each store. The image is displayed on the image display unit 20, or is displayed by blinking on the image display unit 20 so as to be highlighted.

Further, in the second embodiment, the calibration setting unit 167 displays the setting image CG on the image display unit 20a and executes the calibration setting by the index finger FF of the right hand HD detected in the captured image IMG. The setting image CG and the index finger FF can be variously modified. For example, the calibration setting unit 167 causes the image display unit 20a to display an image of one large circle as the setting image CG, and performs the calibration setting based on the displayed circle trajectory and the detected index finger FF trajectory. You may. Further, the specific object detected by the calibration setting unit 167 may be the control unit 10a that is associated with the HMD 100a on a one-to-one basis instead of the index finger FF of the right hand HD, or a ballpoint pen or the like. May be.

G-2. Modification 2:
In the above embodiment, the wireless communication unit 132 of the HMD 100 receives the selection information from the server 300 after the current position of the image display unit 20 is specified by the GPS module 137, but the wireless communication unit 132 of the HMD 100 receives the selection information from the server 300. The transmission/reception method can be variously modified. For example, when the current position of the image display unit 20 is specified and the image of the signboard is detected from the captured image, the wireless communication unit 132 detects the position information of the image display unit 20 and the information of the detected signboard image. May be transmitted to the server 300, and the advertisement information associated with the detected signboard may be received from the server 300. In the HMD 100 of this modification, the advertisement information stored in the server 300 does not have to be stored in the control unit 10 like the selection information storage unit 122 of the storage unit 120 of the above-described embodiment, and thus the storage of the control unit 10 is not necessary. The capacity is small, and the HMD 100 can be miniaturized.

In the above embodiment, the wireless communication unit 132 of the HMD 100 transmits/receives information to/from a server such as the server 300, so that the control unit 10 acquires the image of the signboard and the advertisement information associated with the signboard. The method of acquiring can be variously modified. For example, the storage unit 120 of the control unit 10 stores the image of the signboard and the advertisement information associated with the signboard, and each unit of the control unit 10 advertises using the stored image of the signboard and the advertisement information. Display processing may be performed. The signboard image and the advertisement information may be acquired from an external storage device connected by a USB interface. In the HMD 100 of this modified example, it is not necessary to transmit and receive information to and from other information terminals such as the wireless communication unit 132, so that the HMD 100 can be downsized.

G-3. Modification 3:
In the above embodiment, an example of the display mode of the advertisement image displayed on the image display unit 20 by the display setting unit 165 has been described, but the display mode set by the display setting unit 165 can be variously modified. For example, in the second embodiment, the display setting unit 165 displays the advertisement image TX5 and the like associated with the advertisement image TX4 for which the user has performed the calibration operation near the center of the image display maximum area PN. , BL221, 222 may be increased more than before the operation is performed. When the user performs the calibration operation on the advertisement image TX4 displayed in the image display maximum area PN, it is less necessary to transmit the outside scene SC, and therefore, the brightness of the BLs 221 and 222 is increased to increase the use. The convenience of the person can be improved.

Further, when there are a plurality of detected signboards in the same category in the store-specific advertisement data SD, the display setting unit 165 may display them side by side in the maximum image display area PN so that the user can easily compare them. Further, the display setting unit 165 sets the font of the character image displayed in the image display maximum area PN to the same font for the advertisement images of the same category, or sets the outer frame color of the advertisement image to the same color. May be displayed.

Further, the display setting unit 165 forms a convergence angle by changing the image displayed on the right optical image display unit 26 and the image displayed on the left optical image display unit 28 to allow the user to display on the image display unit 20. The displayed advertisement image may be displayed as a three-dimensional image. For example, the display setting unit 165 may set the distance measured by the distance measuring unit 161 and the position of the advertisement image to be displayed three-dimensionally on the image display unit 20 to the same distance. Also, the display setting unit 165 refers to the stay information VD to display only the advertisement image associated with the detected signboard of the store that is most frequently used by the user in a three-dimensional manner to emphasize the user. May be visually confirmed.

Further, in the above embodiment, the signboard detection unit 168 detects the image of the signboard from the imaged image of the camera 61, but the signboard detection unit 168 can also detect the identification target image from information other than the imaged image of the camera 61. .. For example, the signboard detection unit 168 may detect an image included in the information detected by the infrared sensor, the ultrasonic sensor, or the radar as the identification target image.

G-4. Modification 4:
In the advertisement display process of the first embodiment, various information such as the distance is acquired in the process from step S24 to step S32 in FIG. 7, and the advertisement image is displayed on the image display unit 20 based on the acquired information. However, various kinds of acquired information can be modified in various ways. FIG. 25 is a flowchart of the advertisement display process in the modified example. The advertisement display process shown in FIG. 25 is a process in which the processes from step S24 to step S28 in FIG. 7 are omitted. As described above, in the advertisement display process, the distance between the image display unit 20 and the detected signboard does not have to be measured, environmental information such as temperature does not have to be acquired, and the stay information VD and the detected signboard do not have to be acquired. The relationships do not have to be matched.

Further, in the above embodiment, the advertisement is provided to the image display unit 20 based on the current position of the image display unit 20 specified by the GPS module 137 and the selection information transmitted from the server 300 based on the specified current position. Although the image is displayed, the method of acquiring the advertisement image can be modified in various ways. For example, a transmitting unit such as i-Beacon (registered trademark) that transmits, as optical information, the store-specific advertisement data SD of the store associated with the signboard may be arranged on the signboard instead of the server 300. In this case, the HMD 100 includes a receiving unit that receives the optical information transmitted from the signboard, and the signboard detecting unit 168 detects the image of the signboard included in the optical information received by the receiving unit and the camera 61. The detected signboard may be acquired by matching the signboard with the signboard. Further, the HMD 100 may specify its own position by the wireless communication unit 132 based on the network information acquired via the wireless LAN. Further, the HMD 100 may specify its own position by specifying a characteristic building registered as a landmark included in the captured outside scene. Further, the HMD 100 may identify its own position by receiving information from a server that stores data about an address and a specific intersection via the wireless communication unit 132. In this modification, the position of the image display unit 20 does not need to be acquired by the GPS module 137, which improves the convenience of the user of the HMD 100.

In the above-described embodiment, the detected signboard image is described as an example of the image corresponding to the identification target image in the claims, but the identification target image can be variously modified. For example, the identification target image may be an image such as a sign composed of characters, figures, symbols, and a three-dimensional shape from the captured image, or the shape and pattern of an article having a specific appearance, It may be an image of a combination of colors. Further, in the above embodiment, the advertisement images TX1 and TX2 have been described as examples of the image as the determination information corresponding information in the claims, but the determination information corresponding information and the image as the determination information corresponding information are variously modified. It is possible. For example, the image serving as the determination information correspondence information may be an image of a timetable associated with a signboard at a station, a route map associated with a signboard or a landmark at a bus stop, and a next arrival stop. It may be an image showing related information such as a fare for each stop getting off. Further, the determination information corresponding information may be sound other than the image.

G-5. Modification 5:
In the third embodiment, the past purchase information of the user is given as an example of the personal information of the user stored in the past history storage unit 124b, but the personal information of the user can be variously modified. Examples of the personal information of the user include a Web history searched by the user via the Internet, a history of conversation between the user and other people, and the like.

Further, in the third to sixth embodiments described above, the determination unit 162 estimates the user's intention or action based on the acquired determination information, but the determination unit 162 estimates the user. The operation and the like can be variously modified. The determination unit 162 may infer, for example, a lying state or a seated state as the user's operation state, or the user's moving state such as being in a vehicle or running. You may guess the state. The determination information is information that serves as a trigger for displaying an image such as an advertisement in the image display maximum area PN, and is the signboard described in the above embodiment, the user's past history, external voice, and the like. Information other than the statistical information of the user and the biometric information of the user may be used. For example, the user may be allowed to freely set the judgment information by the operation unit 135 receiving a predetermined operation.

Further, in the above-described fourth embodiment, as the biometric information acquired by the biometric information acquisition unit 39, the sweat of the user is described as an example, but the biometric information acquired by the biometric information acquisition unit 39 can be variously modified. Is. The biological information acquisition unit 39 may acquire, for example, the user's body temperature, blood pressure, heart rate, respiration rate, brain wave, blood flow amount, blood sugar level, and the like. The biometric information acquisition unit 39 is a sensor that acquires these biometric information, and may be included in the HMD 100 either integrally or separately. The HMD 100 of the modification may not include the biometric information acquisition unit 39, and may acquire biometric information of the user acquired by another device via the wireless communication unit 132.

In the sixth embodiment, the determination unit 162 sets the advertisement image to be displayed in the image display maximum area PN based on the external voice acquired by the microphone 63 and the position information of the user. The advertisement image displayed in the image display maximum area PN may be set by external information other than the above. The determination unit 162 may set an advertisement image to be displayed in the image display maximum area PN according to, for example, the image of the outside scene SC, the smell, the weather, etc., in addition to the position information of the user. Therefore, the HMD 100 may include a camera that captures the outside scene SC, an odor sensor that determines an external odor, a temperature sensor that determines the weather, a humidity sensor, an atmospheric pressure sensor, and an ambient light sensor.

The display information associated with the position information in the claims naturally includes, for example, information on a store within a predetermined range from the position of the HMD 100 included in the acquired position information, but may be other information. Good. For example, the acquired position information is information such as stores within a predetermined range from the position where the user of the HMD 100 arrives after a predetermined time (for example, 1 hour) as a position different from the current position of the HMD 100. It may be. In addition, the acquired position information may be information on a tourist facility within a predetermined range from a trajectory that the user of the HMD 100 will pass within a predetermined time (for example, within 30 minutes). Further, the acquired position information may be information that specifies the position of the user of another HMD 100.

Further, the display mode of the image as the determination information corresponding information displayed in the image display maximum area PN is not limited to the example described in the above embodiment, and various modifications are possible. For example, when the display mode of the display image is the display position, the display timing (for example, one image is displayed over a few seconds), the brightness according to the illuminance of the outside scene, the two-dimensional or three-dimensional display, and the three-dimensional display Various image settings such as the distance visually recognized by the user and enlargement or reduction of a specific image can be given.

G-6. Modification 6:
In the above embodiment, the operation unit 135 is formed in the control unit 10, but the mode of the operation unit 135 can be variously modified. For example, there may be a mode in which a user interface, which is the operation unit 135, is provided separately from the control unit 10. In this case, the operation unit 135 is a separate body from the control unit 10 in which the power source 130 and the like are formed, so that the operation unit 135 can be downsized and the operability for the user is improved. Further, although the camera 61 is arranged in the image display unit 20, the camera 61 may be configured separately from the image display unit 20 and can capture the outside scene SC.

For example, the image light generation unit may be configured to include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image light generation unit may use an LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type HMD 100.

Further, for example, the HMD 100 may be a head mount display in which the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit does not completely cover the user's eye. Further, the HMD 100 may be a so-called monocular type head mounted display. Further, although the HMD 100 is described as a binocular type optically transmissive type, the present invention can be similarly applied to other types of head-mounted display devices such as a video transmissive type.

The earphone may be of an ear-hook type or a headband type, and may be omitted. Further, for example, it may be configured as a head-mounted display device mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display device built in a body protection tool such as a helmet.

G-7. Modification 7:
The configuration of the HMD 100 in the above embodiment is merely an example, and can be variously modified. For example, the direction key 16 provided in the control unit 10 may be omitted, or in addition to the direction key 16 and the track pad 14, another operation interface such as an operation stick may be provided. Further, the control unit 10 may have a configuration in which an input device such as a keyboard and a mouse can be connected, and may accept input from the keyboard and the mouse.

Further, as the image display unit, instead of the image display unit 20 worn like glasses, an image display unit of another system such as an image display unit worn like a hat may be adopted. Moreover, the earphones 32 and 34 can be appropriately omitted. Further, in the above embodiment, the LCD and the light source are used as the configuration for generating the image light, but instead of these, another display element such as an organic EL display may be adopted.

FIG. 26 is an explanatory diagram showing the external configuration of the HMD in the modification. In the case of the example of FIG. 26A, the difference from the HMD 100 shown in FIG. 1 is that the image display unit 20x includes a right optical image display unit 26x instead of the right optical image display unit 26, and a left optical image. That is, a left optical image display unit 28x is provided instead of the display unit 28. The right optical image display unit 26x is formed smaller than the optical member of the above-described embodiment, and is arranged diagonally above the right eye of the user when the HMD 100x is worn. Similarly, the left optical image display unit 28x is formed smaller than the optical member of the above-described embodiment, and is arranged diagonally above the left eye of the user when the HMD 100x is worn. In the case of the example of FIG. 26B, the difference from the HMD 100 shown in FIG. 1 is that the image display unit 20y includes a right optical image display unit 26y instead of the right optical image display unit 26, and a left optical image. The point is that a left optical image display unit 28y is provided instead of the display unit 28. The right optical image display unit 26y is formed smaller than the optical member of the above-described embodiment, and is arranged diagonally below the right eye of the user when the head mounted display is worn. The left optical image display unit 28y is formed smaller than the optical member of the above-described embodiment, and is arranged diagonally below the left eye of the user when the head mounted display is worn. Thus, it is sufficient if the optical image display unit is arranged near the user's eyes. Further, the size of the optical member forming the optical image display unit is also arbitrary, the optical image display unit covers only a part of the user's eyes, in other words, the optical image display unit completely covers the user's eyes. It can be realized as the HMD 100 in an uncovered form.

Further, in the above-described embodiment, the HMD 100 may guide image light representing the same image to the left and right eyes of the user to allow the user to visually recognize a two-dimensional image, or the HMD 100 may display different images for the left and right eyes of the user. The image light may be guided so that the user can visually recognize the three-dimensional image.

Further, in the above embodiment, part of the configuration realized by hardware may be replaced with software, and conversely, part of the configuration realized by software may be replaced with hardware. Good. For example, in the above embodiment, the image processing unit 160 and the audio processing unit 170 are realized by the CPU 140 reading and executing the computer program, but these functional units may be realized by hardware circuits. Good.

When some or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the "computer-readable recording medium" is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but various internal RAMs, ROMs, etc. in a computer, a hard disk, etc. It also includes an external storage device that is fixed to the computer.

Further, in the above-described embodiment, as shown in FIGS. 2 and 3, the control unit 10 and the image display unit 20 are formed as separate configurations. However, regarding the configuration of the control unit 10 and the image display unit 20, However, the present invention is not limited to this, and various modifications are possible. For example, all or some of the components formed in the control unit 10 may be formed inside the image display unit 20. Further, the power supply 130 in the above-described embodiment may be formed independently and may be replaceable, or the structure formed in the control unit 10 may be formed in the image display unit 20 in an overlapping manner. For example, the CPU 140 shown in FIG. 2 may be formed in both the control unit 10 and the image display unit 20, or the functions performed by the CPU 140 formed in the control unit 10 and the CPU formed in the image display unit 20. May be separately configured.

The present invention is not limited to the above-described embodiments and modified examples, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments and modified examples corresponding to the technical features in each mode described in the section of the summary of the invention are to solve some or all of the above problems, or In order to achieve some or all of the effects, it is possible to appropriately replace or combine them. If the technical features are not described as essential in this specification, they can be deleted as appropriate.

10... Control part 11... Enter key 12... Lighting part 13... Display switching key 14... Track pad 15... Luminance switching key 16... Direction key 17... Menu key 18... Power switch 20... Image display part 21... Right holding part 22... Right display drive unit 23... Left holding unit 24... Left display drive unit 26... Right optical image display unit 28... Left optical image display unit 30... Earphone plug 32... Right earphone 34... Left earphone 37... Right eye imaging camera 38... Left Eye imaging camera 39... Biological information acquisition unit 40... Connection unit 42... Right code 44... Left code 46... Connecting member 48... Main body code 51, 52... Transmission unit 53, 54... Reception unit 61... Camera (image detection unit)
63... Microphone 65... Thermometer (state detector)
100... HMD
120... Storage unit 122... Selection information storage unit 124... Past history storage unit (personal information storage unit)
130... Power source 132... Wireless communication unit (information transmission/reception unit, state detection unit)
135... Operation unit 137... GPS module (position specifying unit, image detection unit)
140... CPU
150... Operating system 160... Image processing unit 161... Distance measuring unit 162... Judgment unit (judgment information acquisition unit)
165... Display setting unit (display image control unit)
166... Past history collating section 167... Calibration setting section (calibration section)
168... Signboard detection unit (image detection unit, display image control unit)
169... Environmental information processing unit (display image control unit)
170... Audio processing section 180... Interface 190... Display control section 201... Right backlight control section 202... Left backlight control section 211... Right LCD control section 212... Left LCD control section 221... Right backlight 222... Left backlight 241 … Right LCD
242... Left LCD
251... Right projection optical system 252... Left projection optical system 261... Right light guide plate 262... Left light guide plate 300... Server (information storage device)
310... CPU
312... Information selection section 320... Storage section 325... Advertisement DB
330... Communication unit (selection information transmission unit)
500... Information system A, B, C, D... Location VSync... Vertical sync signal HSync... Horizontal sync signal PCLK... Clock signal P1, P2, P3, P4, P5... Circular image OA... External device SC... Outside scene SD... By store Advertising data VD... Stay information HD... Right hand RE... Right eye LE... Left eye FF... Index finger (specific object)
CG... Setting image (specific image)
BG... Square image IL... Illumination light PL... Image light PN... Image display maximum area VR... Field of view AB1, AB2, AB3... Signboard CS1, CS2, CS3, CS4... Arrow image TX1, TX2, TX3, TX4, TX5, TX6 TX7, TX8, TX9, GM... Advertisement image (image as judgment information corresponding information)
CI... Guide image (image as judgment information corresponding information)
IMG...Captured image IC1, IC2, IC3... Icon image IC
GU...Information board FR...Friend

Claims (14)

A transmissive head-mounted display device,
An image display unit that can display an image and can transmit the outside scene,
A position information acquisition unit that acquires position information of the head-mounted display device,
A determination information acquisition unit that acquires an image of a signboard as determination information used to determine whether to display an image displayed on the image display unit,
An information transmitting/receiving unit that is associated with the acquired position information and receives display information including determination information corresponding information,
When the determination information is acquired, a display for displaying the image as the determination information corresponding information included in the received display information, which is associated with the acquired determination information, on the image display unit. An image control unit,
A personal information storage unit for storing personal information;
The display image control unit causes the image display unit to display an image as the determination information corresponding information set based on the personal information of the user,
The personal information includes a central average time as a central average value when the user stayed in the past in each specific place,
The display image control unit, the time difference between the central average time and the current time is smaller than the threshold value, corresponding to the signboard of the image corresponding to specific locations before SL corresponding to the case where the small A head-mounted display device for displaying the image as judgment information corresponding information . The head-mounted display device according to claim 1,
The determination information acquisition unit has an image detection unit that detects an identification target image as the determination information included in the display information from the outside scene,
The head-mounted display device, wherein the display image control unit causes the image display unit to display the determination information corresponding information when the identification target image is detected. The head-mounted display device according to claim 2, wherein
The image detection unit captures the outside scene,
The display image control unit, when the identification target image is detected in the captured outside scene, displays the determination information correspondence information associated with the detected identification target image on the image display unit. Head-mounted display device to display. The head-mounted display device according to claim 2, wherein
The image detection unit specifies the orientation of the image display unit using the position information acquired by the position information acquisition unit and the position information of the signboard that is the determination information,
The display image control unit, when the identification target image is detected in the outside scene based on the identified orientation of the image display unit, the determination associated with the detected identification target image. A head-mounted display device for displaying information corresponding information on the image display unit. The head-mounted display device according to any one of claims 2 to 4,
The judgment information acquisition unit acquires the judgment information based on the signboard,
The identification target image is an image of the signboard,
The head-mounted display device further includes
A position of the head-mounted display device specified by the acquired position information, and a position of the signboard that is the basis of the detected determination information, and a distance measuring unit that measures a distance,
The head mounted display device, wherein the display image control unit sets a display mode of an image as the determination information corresponding information to be displayed on the image display unit based on the measured distance. The head-mounted display device according to claim 5,
The head-mounted display device, wherein the display image control unit sets at least one of an image size and a display position as a display mode of the image as the determination information corresponding information displayed on the image display unit. The head-mounted display device according to claim 5 or 6, further comprising:
The head mounted display device includes a state detection unit that detects a state of an external environment,
The head-mounted display device, wherein the display image control unit causes the image display unit to display an image as the determination information corresponding information associated with the detected state of the external environment. The head-mounted display device according to any one of claims 5 to 7, further comprising:
A specific image is displayed on the image display unit, an image of a specific object in the outside scene is detected, and the image is displayed on the image display unit based on the specific image and the image of the specific object. A calibration unit for calibrating the positional relationship between the image and the object,
The display image control unit, after being calibrated by the calibration unit, based on the position of the image of the specific object detected in the outside scene, the determination information correspondence information displayed on the image display unit. Head-mounted display device that changes the image of the. The head-mounted display device according to any one of claims 5 to 8,
The head-mounted display device, wherein the personal information of the user is information about a place where the user has stayed in the past. The head-mounted display device according to any one of claims 5 to 9,
The judgment information acquisition unit acquires, as the judgment information, at least one of information about a place where the user has stayed in the past, biometric information of the user, and statistical information of other users. Part-mounted display device. The head-mounted display device according to claim 10, further comprising:
Having an eye imaging camera for imaging the user's eyes,
The determination information includes a captured image of the user's eye and a past purchase history of the user's product,
The determination information acquisition unit,
Using the captured image of the eye of the user captured by the eye imaging camera, while specifying the direction of the line of sight of the user, the size of the pupil of the user is specified,
When the size of the pupil becomes larger than a threshold value, a head-mounted display device that determines an image to be displayed on the image display unit using the line-of-sight direction of the user and the past purchase history. .. An information system comprising: an information storage device that stores and transmits information; and a transmissive head-mounted display device that transmits and receives information to and from the information storage device,
The head-mounted display device,
A position information acquisition unit that acquires position information of the head-mounted display device,
An information transmission unit that transmits the acquired position information to the information storage device,
The information storage device,
An information selection unit that selects display information including determination information corresponding information from the information stored in the information storage device based on the position information transmitted from the information transmission unit;
The selected display information, a selection information transmission unit for transmitting to the head-mounted display device,
The head-mounted display device further includes
An image display unit that can display an image and can transmit the outside scene,
A determination information acquisition unit that acquires an image of a signboard as determination information used to determine whether to display an image displayed on the image display unit,
When the determination information is acquired, a display image that is associated with the acquired determination information and causes the image display unit to display an image as the determination information corresponding information included in the received display information. A control unit,
A personal information storage unit for storing personal information;
The display image control unit causes the image display unit to display an image as the determination information corresponding information set based on the personal information of the user,
The personal information includes a central average time as a central average value when the user stayed in the past in each specific place,
The display image control unit, the time difference between the central average time and the current time is smaller than the threshold value, corresponding to the signboard of the image corresponding to specific locations before SL corresponding to the case where the small An information system for displaying the image as judgment information corresponding information . A method for controlling a transmissive head-mounted display device, comprising: an image display unit capable of displaying an image and transmitting an outside scene; an information receiving unit for receiving information; and a personal information storage unit,
Computer
Acquiring position information of the head-mounted display device,
A step of receiving display information which is associated with the acquired position information and includes determination information corresponding information;
A step of acquiring an image of a signboard as judgment information used for determining whether or not to display an image displayed on the image display unit;
When the determination information is acquired, a step of displaying an image as the determination information corresponding information included in the received display information in association with the acquired determination information on the image display unit, ,
Storing personal information in the personal information storage unit,
In the step of displaying, an image as the determination information corresponding information set based on the personal information of the user is displayed on the image display unit,
The personal information includes a central average time as a central average value when the user stayed in the past in each specific place,
Wherein the step of displaying, said if the time difference between the center average time is the current time is smaller than the threshold value, the judgment corresponding to the signboard of the image corresponding to specific locations before SL corresponding to the case where the small A control method comprising the step of displaying the image as information corresponding information on the image display unit. A computer program for a transmissive head-mounted display device having an image display unit capable of displaying an image and transmitting an external scene,
A position specifying function for acquiring position information of the head-mounted display device,
An information transmitting/receiving function that is associated with the acquired position information and receives display information including determination information corresponding information,
A judgment information acquisition function for acquiring an image of a signboard as judgment information used for determining whether to display an image displayed on the image display unit,
When the determination information is acquired, a display image which is associated with the acquired determination information and causes the image display unit to display an image as the determination information corresponding information included in the received display information. Control function,
A personal information storage function that stores personal information, and a computer,
The display image control function causes the image display unit to display an image as the determination information corresponding information set based on the personal information of the user,
The personal information includes a central average time as a central average value when the user stayed in the past in each specific place,
The display image control function, the time difference between the central average time and the current time is smaller than the threshold value, corresponding to the signboard of the image corresponding to specific locations before SL corresponding to the case where the small A computer program for displaying the image as judgment information corresponding information .

Images