JP2016038889A - Extended reality followed by motion sensing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide integration of virtual (for example, calculation) information to a real time image stream on which an environment surrounding a wearer is reflected.SOLUTION: A head mount device 201 comprises an optic assembly 203 for displaying a surrounding environment or virtual environment to a user. When a motion capture system 200 is incorporated to the head mount device 201, the user can control the displayed environment interactively. The system 200 comprises cameras 102 and 104 connected to a sensing processing system 206. For looking a sight of an object area 212 including an object 214 moving in the object area 212, including various kinds of virtual objects 216 and virtually drawn or virtually extended, the cameras 102 and 104 are directed to part of the object area 212 by movement of the device 201. One or more sensors 208 and 210 grasp movement of the device 201.SELECTED DRAWING: Figure 2

Description

  The disclosed technology is for use in a wearable sensing system that can detect gestures in a three-dimensional (3D) sensing space using an image sensor or other sensor and present 3D augmented reality to a user. The present invention relates to a high-performance and high-precision sensing and imaging apparatus.

 Classification devices such as Google Glass provide the ability to superimpose and present information on a see-through screen worn by the user. Other classes of devices, such as Oculus Rift, provide a virtual reality display for users without information from the real world surrounding the user. However, both classes of devices cannot accurately provide integration of virtual (eg, computational) information into a real-time image stream that reflects the environment surrounding the wearer. Therefore, there is a need for a sophisticated sensing and imaging device that can capture scene image information and provide the user with at least near-real-time image information pass-through. It is said. The sensing and imaging device is ideally a wearable device or portable device that creates a wearable sensing system that can present expanded image information to the wearer along with the presentation of virtualized or created information. It can be connected to the device. No device has been known to provide this capability.

  Implementation of the disclosed technology addresses the above and other issues by providing a motion sensing and imaging device that can capture landscape image information and provide a user with at least near real-time image information pass-through. deal with. The motion sensing and imaging device is connected to a wearable device or a portable device that creates a wearable sensing system that can present expanded image information to the wearer alone or with virtualized or created information. Used.

  In one embodiment of the motion sensing and imaging apparatus, the apparatus comprises a plurality of image sensors arranged to provide stereoscopic image information of the scene being viewed, and 1 arranged around the image sensor. Alternatively, a plurality of illumination sources and a controller connected to the image sensor and the illumination source to control operations of the image sensor and the illumination source are provided. The controller enables the device to obtain the landscape image information and provide the user with a pass-through of at least approximately real-time image information. The device can be connected to a wearable device to create a wearable sensing system that can present expanded image information to the wearer along with virtualized or created information.

  In one embodiment, the motion sensing and imaging device captures image information for control objects (including control objects such as human hands) within the visible range of the image sensor. The image information for the controlled object is used to determine gesture information that indicates a command for the machine under control. In an embodiment, the device detects the position, posture and movement of objects around the wearer of the device with submillimeter accuracy and integrates this information into the presentation provided to the wearer. Can be provided for.

  In one embodiment, the motion sensing and imaging device separates information received from pixels sensitive to infrared light from information received from pixels sensitive to visible light, eg, RGB (red, green, blue). Process information from IR (infrared) sensors for use in gesture recognition, and provide information from RGB sensors to provide as a live video feed via a presentation interface. Can be processed. For example, a video stream containing a series of images of a real world landscape is captured using a camera with a set of RGB pixels and a set of IR (infrared) pixels. Information received from pixels that are sensitive to infrared light is separated for processing to recognize a gesture. Information received from RGB sensitive pixels is provided to the presentation interface of a wearable device (HUD, HMD, etc.) as a live video supply to the presentation output. The presentation output is displayed to the user of the wearable device. One or more virtual objects may be integrated with the video stream image to form the presentation output. Thus, the device can provide either gesture recognition, real-world presentation of real-world objects via pass-through video supply, and / or augmented reality including virtual objects integrated with real-world views. is there.

  In one embodiment, a motion sensing and imaging device can be used to detect the motion of the device itself using a combination of RGB and IR pixels of the camera. In particular, this can be achieved by using RGB pixels to obtain global or coarse features of the real world space and corresponding feature values, and using IR pixels to obtain fine or accurate features of the real world space. And obtaining the corresponding feature value. Once obtained, motion information of the wearable sensing system for at least one feature of the landscape is determined based on a comparison of feature values detected at different time instances. For example, the feature of the real world space is an object at a predetermined position in the real world space, and the feature value may be a three-dimensional (3D) coordinate of the position of the object in the real world space. If position coordinate values change between several pairs of image frames or other image volumes, this is used to determine the motion information of the wearable sensing system for objects whose position has changed between image frames. it can.

  As another example, the feature of the real world space is a wall in the real world space, and its corresponding feature value is the direction of the wall sensed by a viewer operating in a wearable sensing system. In this example, if a change in the direction of the wall is registered between successive image frames captured by a camera that is electrically connected to the wearable sensing system, this means that the wearable looking at the wall. It can be said that it shows a change in the position of the sensing system.

  Depending on the embodiment, the information from the RGB pixels of the camera can be used to capture an image or series of objects such as contours, shapes, volume models, skeletal models, silhouettes, overall arrangements, and / or structures of objects in the real world. It can be used to identify objects in the real world along with the prominent or overall characteristics of the object obtained from the image. This can be achieved by measuring the average pixel intensity of the region or the change in region properties. That is, a rough evaluation of the real world or an object in the real world can be obtained by the RGB pixels.

  In addition, data from IR pixels can be used to capture fine or accurate features of real world space, highlighting data extracted from RGB pixels. Examples of fine features include real world space and surface properties, edges, curvatures, and other faint features of objects in real world space. In one embodiment, RGB pixels capture a solid model of the hand while IR pixels are used to capture the vein and / or arterial pattern or fingerprint of the hand.

  Some other implementations include capturing image data using RGB pixels and IR pixels in different combinations and permutations. For example, one embodiment includes activating RGB pixels and IR pixels simultaneously and performing a collection of image data on a global scale without distinguishing between coarse and fine features. Other embodiments include the intermittent use of RGB and IR pixels. Still other embodiments include activating RGB and IR pixels based on quadratic or Gaussian functions. Some other implementations include performing a scan with IR pixels first, followed by a scan with RGB pixels, and performing the scan in the reverse order.

  In one embodiment, ambient lighting conditions are determined and can be used to adjust the display of the output. For example, information from the RGB pixel set is displayed under normal lighting conditions, and information from the IR pixel set is displayed under dark lighting conditions. Alternatively or in addition, information from the IR pixel set can be used to enhance information from the RGB pixel set in low light conditions and vice versa. In some implementations, a selection is received from the user indicating a preferred display selected from one of the color images from the RGB pixels and the IR image from the IR pixels, or a combination thereof. Furthermore, as an alternative or addition, the device itself has captured using RGB sensitive pixels for display depending on ambient conditions, user selection, situational awareness, other factors, or combinations thereof. Video information and video information captured from pixels sensitive to infrared light may be dynamically switched.

  In one embodiment, information from pixels that are sensitive to infrared light is separated for processing to recognize gestures, while information from pixels that are sensitive to RGB is provided to the output as a live video feed, As a result, bandwidth for gesture recognition can be secured. In gesture processing, it is possible to detect image features corresponding to objects in the real world. The object features that are correlated with gesture motion are correlated across multiple images to determine changes. Gesture motion can be used to determine command information for a machine under control, an application embedded therein, or a combination thereof.

  In one embodiment, a motion sensor and / or other type of sensor is connected to the motion capture system so that the motion capture system monitors at least the operation of the sensor of the motion capture system due to, for example, user contact. To do. Information from the motion sensor can be used at first and second times to determine first and second position information of the sensor relative to a fixed point. Difference information between the first and second position information is determined. Motion information about the device relative to a fixed point is calculated based on the difference information. Movement information about the device is provided in the apparent environmental information detected by the sensor, and the actual environmental information is obtained by subtracting the movement of the sensor from the apparent environmental information. Information can be communicated. Control information may be generated from a sensor configured to provide a virtual or augmented reality experience via a portable device and / or in a space tuned to remove the movement of the sensor itself. Can be communicated to a system that controls the duration and the like based on motion capture information about the moving object. In some applications, haptic, audio, and / or visual projectors can be added to extend the virtual device experience.

  In one embodiment, the apparent environmental information is obtained from the position information of the part of the object at the first and second times using the sensors of the motion capture system. The movement information of the object part relative to the fixed point at the first and second times is calculated based on the difference information and the movement information of the sensor.

  In a further embodiment, in a series of times, the motion information of the sensor is determined using a motion sensor, the part of the object is repeatedly determined using the sensor, and the path of the part of the object relative to the fixed point is determined. In order to do this, the path of the object is calculated by analyzing a series of motion information. The path can be contrasted with a template for identifying the trajectory. The locus of the body part is identified as a gesture. A gesture can point to command information communicated to the system. Some gestures convey commands that change the operating mode of the system (eg, zoom in, zoom out, pan, show more detail, next display page, etc.).

  Some embodiments advantageously allow for improved user experience, greater safety, and improved functionality for wearable device users. Some implementations further provide the motion capture system with the ability to recognize gestures by allowing a user to perform intuitive gestures with virtualized contact with virtual objects. For example, the device is provided with the ability to distinguish object motion from the device's own motion to facilitate accurate gesture recognition. In some embodiments, various portable or wearable machines (eg, smartphones, portable computing systems including laptop computers, tablet computing devices, personal data assistants, eg, head-ups used in airplanes and automobiles) Application-specific visualization computing mechanisms including displays (HUD), wearable virtual and / or augmented reality systems such as Google Glass, graphics processors, embedded microcontrollers, gaming consoles, etc., and one or more of these wires Or an improved interface with a wirelessly coupled network and / or a combination thereof, such as a mouse, joystick, touchpad, or touchscreen connection. The need for the type of input device can be removed or reduced. Some embodiments may provide a further improved interface for computing and / or other mechanisms than was possible with the prior art. In some implementations, a richer human machine interface may be provided.

  Other aspects and advantages of the technology will become apparent from the following drawings, detailed description, and claims.

1 is a diagram illustrating an example of a motion sensing and imaging apparatus.

1 illustrates one embodiment of a wearable sensing system based on motion sensing and imaging devices. FIG.

1 is a simplified block diagram of a computer system.

The figure which shows the basic operation and functional unit accompanying a motion capture and image analysis.

FIG. 5 is a diagram illustrating an example of augmented reality pass-through presented by a device capable of motion sensing and imaging.

  The disclosed technique captures true or near real-time scenery, detects gestures in the 3D sensing space, interprets the gestures as commands to the controlled system or machine, and captures them where appropriate. The present invention relates to a motion sensing and imaging apparatus capable of providing image information and the command.

  Embodiments allow a user of a virtual reality device to “pass-through” in which live video is provided, allowing the user to perceive the real world directly, or one or more virtual objects Providing with the display. For example, the user can see the actual desk environment as well as the virtual application or objects mixed there. Gesture recognition and sensing gives the user the ability to grip or interact with a real object (eg, a user's cola can) alongside a virtual object (eg, a virtual document that floats above the surface of the user's actual desk). Allows the embodiment to be provided. In some implementations, information from different spectral sources is selectively used to drive one or other aspects of the experience. For example, information from a sensor that is sensitive to infrared light can be used to detect a user's hand movement and recognize a gesture. On the other hand, information from the visible light region can be used to create real world presentations of real and virtual objects to drive pass-through video presentations. As a further example, a combination of image information from multiple sources can be used so that the system or user can select an infrared image and a visible light image based on the situation, condition, environment, other factors, or a combination thereof. Select. For example, the device can switch from a visible light image to an infrared image if the ambient light conditions are guaranteed. It is also possible for the user to have the ability to control the imaging source as well. As yet another example, information from one type of sensor can be used to extend, correct, or augment information from another type of sensor. Information from the infrared sensor can be used to correct a derived image display from a sensor that is sensitive to visible light, and vice versa. Detect audio signals or vibration waves to provide object direction and position in low light or other situations where an optical image cannot be derived, i.e. in situations where free-form gestures cannot be optically recognized with sufficient confidence Can be used to This point will be further described.

  The disclosed technology can be applied to improve the user experience in an immersive virtual reality environment using a wearable sensing system. Examples of systems, devices, and methods according to the disclosed embodiments are described in the context of a “wearable sensing system”. Examples of “wearable sensing systems” are provided solely for the purpose of adding context and clues in understanding the disclosed technology. In other examples, gesture-based interaction examples in other contexts such as cars, robots, or other machines may be applied to virtual games, virtual applications, virtual programs, virtual operating systems, and the like. While other applications are possible, the examples given below should not be treated as definitive or restrictive in any of the scope, circumstances or settings. Thus, it will be apparent to those skilled in the art that embodiments can be implemented within and outside the context of a “wearable sensing system”.

  First, refer to FIG. 1 showing an example of a motion sensing device 100. As shown in FIG. 1, the motion sensing device 100 includes a lighting board 172 that can be coupled to the main board 182 by a screw fixing component or other method. Cable routing (not shown in FIG. 1 for clarity) provides electrical connection between the lighting board 172 and the main board 182 to allow signal and power exchange. The fixing component for joining the main board 182 (first part) and the lighting board 172 (second part) is further connected to an electronic device (for example, HMD, HUD, smartphone) of a wearable type or a portable type. It can be fixed to the mounting surface A. The attachment surface A can be a surface (internal or external) of a wearable or portable electronic device. Furthermore, the device may be installed in a cavity or container of a wearable or portable electronic device by fitting, screwing, or a combination thereof. The device 100 can be incorporated into any of a variety of wearable or portable electronic devices to meet a wide range of application design requirements.

  The illumination board 172 has a number of individually controllable illumination sources 115, 117 which are LEDs or other light sources incorporated on the board. In the illustrated embodiment, two cameras 102, 104 are provided on the main board 182 of the device 100 to provide stereoscopic image type detection of the scene being viewed. The main board 182 further includes a processor that performs basic image processing and controls the LEDs of the cameras 102 and 104 and the board 172. Various modifications can be made to the design shown in FIG. For example, the number and arrangement of LEDs, photodetectors, and cameras may be changed. Also, the scanning and imaging hardware may be integrated on one board. Alternatively, these changes can be made as appropriate according to the needs of a specific application.

  The stereoscopic image information provided by the cameras 102 and 104 is provided selectively or continuously to a wearable or portable electronic device worn or carried by the user. The device 100 may be extended in real time or near real time, such as by providing raw “real time” or near real time image information, computer generated graphics, information, icons, or other virtualized presentations. It is possible to provide temporal image information, a virtualized display of the scenery being viewed, and time-varying combinations selected from these. User gestures are detected by the cameras 102, 104 of the sensing device 100, and the resulting image information confirms and determines instructions for any system (including wearable or portable devices) controlled from the gesture. Can be provided for motion capture. By integrating gesture recognition with imaging capability into a single motion sensing device 100, it is highly functional, adaptable, and small enough to be installed in a wearable or portable electronic device. An apparatus can advantageously be provided.

  In some embodiments, some of the illumination sources 115, 117 can have associated focusing optics (not shown in FIG. 1 for clarity). Either of the boards 172, 182 may further include a socket for coupling to a photodetector (or other sensor) not shown in FIG. 1 for clarity. The information obtained by detecting the change in reflectance by the light detector is the presence or absence of an object in the spatial region where the illumination source emits light while the LED, for example, the LED scans the spatial region. Indicate.

  FIG. 2 illustrates a system 200 for capturing image data in accordance with one embodiment of the disclosed technology. System 200 is preferably connected to wearable device 201. The wearable device 201 may be a head mounted display (HMD) having a Google form factor and a helmet form factor as shown in FIG. 2, or may be incorporated in or connected to a watch, a smartphone, or other portable device. Thus, a wearable sensing system can be formed.

  In various embodiments, the systems and methods described herein for capturing 3D motion of objects can be integrated with other applications such as head mounted devices or portable devices. As shown in FIG. 2, the head mount device 201 may include an optical assembly 203 that displays a surrounding environment or a virtual environment to the user. By incorporating the motion capture system 200 in the head mount device 201, the user can interactively control the displayed environment. For example, the virtual environment can include a virtual object that can be manipulated with a user's hand gesture tracked by the motion capture system 200. In one embodiment, the motion capture system 200 integrated with the head mount device 201 detects the position and shape of the user's hand so that the user can see the hand and interactively control objects in the virtual environment. Then, the detected hand is projected onto the display of the head mount device 201. This can be applied to games and Internet browsing, for example.

  The system 200 includes cameras 102 and 104 that connect to a sensing processing system 206. Cameras 102 and 104 may be any type of camera, including cameras that are sensitive to the visible light spectrum, or cameras that have enhanced sensitivity in a limited wavelength range (eg, infrared or ultraviolet). Also, more generally, the term “camera” refers here to any device (or combination of devices) that can capture an image of an object and display the image in digital data format. For example, instead of a conventional camera that captures a conventional two-dimensional (2D) image, a line sensor or a line camera may be employed. The term “light” generally means any electromagnetic radiation, which may or may not be visible light, broadband light (eg, white light) or narrowband light (eg, single wavelength light, Narrow band wavelength light).

  Cameras 102 and 104 preferably have the ability to capture video images (ie, image frames that are continuous at a substantially constant rate, such as about 15 frames per second). However, it is not limited to a specific frame rate. The capabilities of the cameras 102, 104 are not critical to the disclosed technology. The camera can be changed in terms of frame rate, image resolution (eg, number of pixels per image), color or intensity resolution (eg, bits of intensity data per pixel), lens focal length, depth of field, etc. It is. In general, any camera that can focus on a subject within the spatial size of interest can be used for a particular application. For example, in order to capture the movement of a person's hand other than the hand, the size can be defined by a cube of about 1 meter on a side.

  As shown, the object 214 (in this example, one or more hands) moving within the object region 212 is included, and is virtually depicted or virtually expanded to include various virtual objects 216. In order to view the scene of the target area 212, the cameras 102 and 104 are directed to a part of the target area 212 by the movement of the apparatus 201. One or more sensors 208, 210 capture the movement of the device 201. In some embodiments, one or more illumination sources 115, 117 are arranged to illuminate the region of interest 212. In some embodiments, one or more cameras 102, 104 are placed opposite the motion to be detected, for example, where the hand 214 is expected to move. The amount of information recorded for the hand is proportional to the number of pixels the hand occupies in the camera image, and the closer the camera angle to the hand's “pointing direction” is to a right angle, the more pixels the hand occupies. The above position is the best position. For example, a sensing processing system 206 such as a computer system controls the operation of the cameras 102 and 104 to capture an image of the target area 212 and controls the operation of the sensors 208 and 210 to capture the movement of the apparatus 201. Can do. Information from the sensors 208, 210 is applied to the model of the image acquired by the cameras 102, 104 to offset the effects of movement of the device 201 and is more accurate for the virtual experience depicted by the device 201. Is provided. Based on the captured image and the movement of the device 201, the sensing processing system 206 determines the position and / or movement of the object 214 and renders their display to the user via the assembly 203.

 For example, as an action to determine the movement of the object 214, the sensing processing system 206 determines which pixels of the various images captured by the cameras 102, 104 contain portions of the object 214. In some implementations, any pixel in the image is classified as either an “object” pixel or a “background” pixel, depending on whether the pixel includes a portion of the object 214. Therefore, the object pixel can be easily distinguished from the background pixel based on the brightness. Furthermore, the edge of the object is easily detected based on the difference in brightness between adjacent pixels, and the position of the object in each image can be determined. In some implementations, silhouettes of objects are extracted from one or more object images to reveal information about objects visible from different viewpoints. In some implementations, the silhouette can be obtained using many different techniques, but the silhouette is obtained by using a camera that captures the image of the object and analyzing the image to detect the edge of the object. By correlating the object position between the images of the cameras 102 and 104 and canceling the movement of the device 201 obtained from the sensors 208 and 210, the sensing processing system 206 can determine the position of the object 214 in 3D space. Thus, analyzing the series of images allows the sensing processing system 206 to reconstruct the 3D motion of the object 214 using existing motion algorithms or other techniques. For example, US application 13 / 414,485 (filed March 7, 2012), US provisional application 61 / 724,091 (filed November 8, 2012), and US provisional application 61 / 587,554 (2012). (Filed Jan. 17). All the disclosures are incorporated herein by reference.

  The presentation interface 220 is a virtual (or virtualization) created by an application that can be loaded into or collaboratively implemented in the optical assembly 203 of the device 201 to provide a personal virtual experience to the user of the device. In order to display (real, rendered) objects (visual, audio, tactile, etc.), a projection technique is employed along with sensor-based tracking. The projection can include an image of the object or other visual display.

  In one implementation, motion sensors and / or other types of sensors are used to monitor motion in the real environment. Virtual objects that are integrated within an expanded depiction of the real environment are projected to the user of the portable device 201. The motion information of the user's body part can be determined based at least in part on sensing information received from the image sensors 102, 104 or an acoustic sensor or other sensor device. Control information is communicated to the system based at least in part on a combination of motion of the portable device 201 and user motion determined from sensing information received from the image sensors 102, 104 or acoustic sensors or other sensor devices. . The experience with the visual device can be extended in some embodiments by tactile, audio, and / or other sensing information projectors. For example, referring to FIG. 5, a visual projection assembly 504 may include a page image (eg, a virtual book object superimposed on a desk 216, eg, a real-world object displayed to a user via a live video feed (eg, A virtual device 501) can be projected. The result is a virtual device experience of reading a real book or an electronic book on a physical electronic reader, even if no book or electronic reader is present. The tactile projector 506 can project the feel of the “virtual paper” property of the book onto the reader's finger. The audio projector 502 detects a reader's swipe operation for turning a page and projects a page turning sound. Since it is an augmented reality world, the back side of the hand 214 is projected to the user, and the user sees the scenery as if the user is looking at his hand.

  Referring again to FIG. 2, a plurality of sensors 208, 210 are connected to the sensing processing system 206 to capture the motion of the device 201. Sensors 208 and 210 may be any sensors useful for obtaining signals from various parameters of motion (acceleration, velocity, angular acceleration, angular velocity, position / location). More generally, the term “motion detector” refers to any device (or combination of devices) that has the ability to convert mechanical motion into an electrical signal. The device includes an accelerometer, a gyroscope, and a magnetometer, alone or in various combinations, and is designed to detect motion through changes in direction, magnetism, or gravity. There are many types of motion sensors, and alternative embodiments are varied.

  The illustrated system 200 may be any of a variety of other sensors, not shown in FIG. 2 for clarity, either alone or in various combinations to enhance the virtual experience provided to the user of the device 201. Can be included. For example, the system 206 switches to a touch mode in which a touch (contact) gesture is recognized based on an acoustic or vibration sensor in a low light situation where a freeform gesture cannot be optically recognized with sufficient reliability. Alternatively, when a signal from an acoustic or vibration sensor is detected, the system 206 may switch to the touch mode or perform a touch detection process and a complementary image capture process. Further, as another mode of operation, the tap or touch gesture behaves as a “wake-up” signal, causing the image and sound analysis system 206 to transition from the standby mode to the mode of operation. For example, the system 206 can transition to the standby mode when there is no optical signal from the cameras 102 and 104 for longer than the threshold interval.

  Of course, the matter shown in FIG. 2 is an example. In some embodiments, it is also preferred that the system 200 be housed in a differently shaped housing, or incorporated within a larger component or assembly. Further, the number and type of image sensors, motion detectors, illumination sources, etc. are illustrated for clarity and the size or quantity is not the same in all embodiments.

  Next, a description will be given with reference to FIG. FIG. 3 is a simplified block diagram of a computer system 300 for implementing the sensing processing system 206 in accordance with an embodiment of the present technology. The computer system 300 includes a processor 302, a memory 304, a motion detector and camera interface 306, a presentation interface 220, a speaker 309, a microphone 310, and a wireless interface 311. The memory 304 is used to store instructions executed by the processor 302 and input and / or output data associated with the execution of the instructions. In particular, the memory 304 is conceptually described as a group of modules (details will be described later) and stores instructions that control the operation of the processor 302 and its interaction with other hardware components. An operating system (OS) oversees basic system functions at a low level, such as memory allocation, file management, and mass storage operation. The operating system is configured by various operating systems listed below, or includes the operating system. Examples of the various operating systems include Microsoft's WINDOWS (registered trademark) operating system, UNIX (registered trademark) operating system, Linux (registered trademark) operating system, Xenix operating system, IBM's AIX operating system, and Hewlett-Packard. UX operating system, Novell's NETWARE operating system, Sun Microsystems' SOLARIS operating system, OS / 2 operating system, BeOS operating system, MACINTOSH OS operating system, APACHE operating system, OPENACTION operating system Temu, mobile operating systems such as iOS Android, or or other operating system or platform is assumed.

  The computing environment may further comprise other removable / non-removable, volatile / nonvolatile computer storage media. For example, a hard disk drive can read and write a non-removable nonvolatile magnetic storage medium. The magnetic disk drive can read from or write to a removable non-volatile magnetic storage medium, and the optical disk drive can read from an optical disk such as a removable non-volatile CD-ROM, Alternatively, writing to the same optical disk is possible. Other removable / non-removable, volatile / nonvolatile computer storage media that can be used in an exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, and digital versatile discs (DVDs). Digital video tape, semiconductor RAM, semiconductor ROM, and the like. The storage medium is typically connected to the system bus via a removable / non-removable memory interface.

  The processor 302 may use a general-purpose microprocessor, but in some embodiments, alternatively, a microcontroller, peripheral integrated circuit elements, CSIC (customer-specific IC), ASIC (application-specific IC), logic circuit, digital Programmable logic devices such as signal processors, FPGAs (Field Programmable Gate Arrays), PLDs (Programmable Logic Devices), PLAs (Programmable Logic Arrays), RFID processors, smart chips, or any other device, or this technology Arrangement of devices that can execute the above processing operations can be used.

  The motion detector and camera interface 306 includes hardware and / or software that enables communication between the computer system 300 and the cameras 102 and 104 as well as the sensors 208 and 210 (see FIG. 2). Thus, for example, the motion detector and camera interface 306 may receive data from the camera and motion detector before being provided with signals as input to a motion capture (“mocap”) program 314 that is executed by the processor 302. Similar to hardware and / or software signal processors that modify signals (eg, noise reduction, data reformatting, etc.), cameras, illumination sources, and motion detectors (via conventional plugs and jacks) One or more connectable camera data ports 316, 318, illumination source ports 313, 315, and motion detector ports 317, 319. In some embodiments, the motion detector and camera interface 306 further sends signals to the camera, illumination source, and sensor, eg, camera settings (frames to activate or deactivate them). Rate, image quality, sensitivity, etc.), illumination source settings (intensity, illumination time, etc.), sensor settings (calibration, sensitivity level, etc.), etc. can be controlled. The signal is transmitted in response to a control signal from the processor 302, for example. Note that control signals from the processor 302 may be generated in sequence in response to user input or other detected events.

  Instructions defining the mocap program 314 are stored in the memory 304, and when executed, the instructions are images supplied from the camera and audio signals supplied from sensors connected to the motion detector and camera interface 306. Execute motion capture analysis for. In one embodiment, the mocap program 314 includes various modules such as an object analysis module 322 and a path analysis module 324. The object analysis module 322 may analyze the image to detect object edges and / or other information regarding the position of the object in the image (eg, an image captured via the interface 306). it can. In some implementations, the object analysis module 322 may include an audio signal (e.g., via the interface 306) for localization of the object, e.g., for the time distance (time required) of arrival of the object, multilateration, etc. Audio signals captured in this manner can also be analyzed. Multilateration is a navigation technique based on measurement of a distance difference between two or more bases that transmit signals from a known position at a known time (for example, Wikipedia: http: // en .wikipedia.org / w / index.php? title = Multilateration & oldid = 523281858, on Nov. 16, 2012, 06:07 UTC)). The path analysis module 324 can track and predict the motion of the object in 3D based on information obtained through the camera. In some implementations, the virtual reality / augmented reality environment manager 326 is similar to the composite object 216 for presentation to the user of the device 201 via the presentation interface 220 that provides a personal virtual experience 213. Providing integration of virtual objects that reflect real objects (eg, hand 214). One or more applications 328 extend or customize the functionality of the device 201 to load into the memory 304 (or otherwise available to the processor 302) for the system 200 to function as a platform. be able to. A series of camera images is analyzed at the pixel level to extract object motion and velocity. The audio signal identifies an object on a known surface and the intensity and change of the signal can be used to detect the presence of the object. If both audio and image information are available at the same time, both types of information can be analyzed and adjusted to provide a more detailed and / or accurate path analysis. Video supply integrator 329 provides integration of live video supply from cameras 102 and 104 and one or more virtual objects (eg, 501 in FIG. 5). Video supply integrator 329 manages the processing of video information from different types of cameras 102, 104. For example, information received from pixels that are sensitive to infrared light and pixels that are sensitive to visible light (eg, RGB) may be separated by the integrator 329 and processed differently. Image information from the IR sensor can be used for gesture recognition, while image information from the RGB sensor can be provided as a live video feed via the presentation interface 220. Information from one type of sensor can be used to enhance, correct, and / or augment information from another sensor. Information from one type of sensor may be advantageous in several types of situations or environmental conditions (eg, low light, fog, bright light, etc.). The apparatus can select whether to provide presentation output based on one or other types of image information automatically or to accept and provide selection from the user. The integrator 329, together with the VR / AR environment manager 326, controls the creation of the environment presented to the user via the presentation interface 220.

  Presentation interface 220, speaker 309, microphone 310, and wireless interface 311 can be used to facilitate interaction with computer system 300 via user device 201. These components can generally be of conventional design or modified as desired to provide any type of user interaction. In some implementations, the results of motion capture using the motion detector and camera interface 306 and the mocap program 314 are interpreted as user input. For example, hand gestures or movements across the surface that are analyzed using the mocap program 314 can be performed, and the results of the analysis can be obtained from other programs running on the processor 302 (eg, web browser, word processor, or other Can be interpreted as a command to the application. That is, by way of illustration, the user can use the gesture from the speaker 309 to move his hand up or down to “scroll” the web page currently displayed to the user of the device 201 via the presentation interface 220. In order to increase or decrease the volume of the audio output, a gesture that turns the hand may be used. The path analysis module 324 displays the detected path as a vector and extrapolates to predict the path, for example, to predict motion and improve the depiction of actions on the device 201 by the presentation interface 220. can do.

  It is understood that the computer system 300 is an example, and various modifications and changes can be made. The computer system can be realized in various form factors such as a server system, a desktop system, a laptop system, a tablet, a smartphone, or personal digital assistants. Certain implementations may also include other functions not described herein, such as, for example, wired and / or wireless network interfaces, media playback and / or recording functions. In some embodiments, one or more cameras and one or more microphones may be incorporated into the computer rather than supplied as separate components. Furthermore, the image or audio analyzer is suitable for part of a computer system component (for example, a processor that executes program code, an ASIC, or a digital signal processor with a fixed function, and receiving image data and outputting analysis results). (I / O interface) alone.

  Although the computer system 300 has been described with reference to specific blocks, the blocks are defined for convenience of description and are not intended for a specific physical arrangement of components. Further, the blocks need not necessarily correspond to physically separate components. When physically separate components are used, connection between the components (for example, connection for data communication) can be wired and / or wirelessly connected according to the request. Thus, for example, due to the execution of the object analysis module 322 by the processor 302, images and / or audio data are analyzed to move across the plane for detecting the entrance of the object, and the image of the contacting object and The processor 302 may operate the motion detector and camera interface 306 to capture audio signals.

  FIG. 4 illustrates the basic operations and functional units 400 involved in motion capture and image analysis according to an embodiment of the present technology. As shown in FIG. 4, the cameras 402 and 404 record a digital image 410 of the scenery. Each digital image is captured by an associated camera image sensor as an array of pixel values, and the digital image remains in raw data, or following conventional preprocessing, one or more frames. It is transferred to the buffer 415. The frame buffer is a volatile memory partition or dedicated segment that stores a “bitmap” image frame 420 corresponding to the pixel value of the image as output by the camera that recorded the image. The bit map is generally configured as a grid conceptually, and each pixel is mapped to an output element of the display device in a one-to-one or other manner. However, it should be emphasized that the topology of how the memory cells are physically organized within the frame buffer 415 is not important for conceptual organization and does not need to match directly.

  The number of frame buffers included in the system generally reflects the number of images that are analyzed simultaneously by the analysis system or analysis module 430 (details will be described later). Briefly, the analysis module 430 analyzes each pixel data of a series of image frames 420 in order to place objects there and track their movement over time (indicated by reference numeral 440). . This analysis can take a variety of forms, and the algorithm that performs the analysis indicates how to treat the pixels of the image frame 420. For example, the algorithm executed by the analysis module 430 can process the pixels of each frame buffer line by line. That is, each row of the pixel grid is analyzed sequentially. Other algorithms analyze pixels in rows, tiled regions, or other organized formats.

  In various embodiments, the motion captured in the series of camera images is used to calculate a corresponding corresponding output image for display on the display 220. For example, a camera image of a moving hand can be converted by the processor 302 into a wireframe display or other graphic display of the hand. Alternatively, hand gestures can be interpreted as input used to control separate visual outputs. Illustratively, the user can move his hand up or down to scroll through the currently displayed web page or other document, or open his hand to zoom in or out of the web page. Close or use gestures. In any case, the output image is typically stored in pixel data format in one of the frame buffers, eg, frame buffer 415. The video display controller reads the frame buffer and generates a data stream and associated control signals for outputting an image to the assembly 203. The video display controller provided by the presentation interface 220 can be provided on the motherboard of the computer 300 along with the processor 302 and the memory 304, and can be integrated with the processor 302 or can operate a separate video memory. Can be implemented as a coprocessor. As described above, the computer system 300 can include a separate graphics or video card that helps generate the supply of output images to the assembly 203. In one embodiment, video cards typically include a graphics processing unit (GPU) and video memory, and are particularly useful for complex and computationally expensive image processing and rendering. The graphics card can include frame buffer and video display controller functionality (the on-board video display controller can be disabled). In general, the image processing and motion capture functions of the system can be distributed between the GPU and the main processor 302 in various ways.

  Algorithms suitable for the motion capture program 314 are described below, but more specifically, for example, January 17, 2012, March 7, 2012, November 8, 2012, December 21, 2012, and US application 61 / 587,554, 13 / 414,485, 61 / 724,091, 13 / 724,357 and 13 / 742,953 filed on January 16, 2013, respectively. All the disclosures are incorporated herein by reference. Various modules include, for example, C, C ++, C #, OpenGL, Ada, Basic, Cobra, FORTRAN, Java (registered trademark), Lisp, Perl, Python, Ruby, or Object Pascal, or a low-level assembly language. Can be programmed in any suitable programming language, including but not limited to high level languages.

  Referring to FIG. 4 again, the operation mode of the apparatus provided in the motion sensing control apparatus is to register the roughness of the data provided to the image analysis module 430, the roughness of the analysis, or both in the registration of the performance database. Based on the determination. For example, during wide mode operation, the image analysis module 430 can operate on all image frames, and each data in the frame buffer 415 is a series of data on all data within the frame capacity limit. When configured as a line, it is possible to instruct the analysis of the reduction amount (that is, resolution) of image data per frame, or discard some frames. The technique by which data is missing from the analysis can depend on the image analysis algorithm or use made with the addition of motion capture output. In some implementations, data is dropped symmetrically or uniformly, such as every other line, every other line, etc., and discarded to the acceptable limit of the application using the image analysis algorithm or its output. In other embodiments, the frequency of line loss increases towards the end of the frame. Still other image acquisition parameters that can be varied include frame size, frame resolution, and the number of frames acquired per second. In particular, the frame size can be reduced, for example, by discarding edge pixels, resampling at a lower resolution (and using only a portion of the frame buffer capacity), and the like. Parameters related to the acquisition of image data (eg, size, frame rate, and characteristics) are collectively referred to as “acquisition parameters”, while the operation of the image analysis module 430 (eg, defining the contour of an object). Related parameters are collectively referred to as “image analysis parameters”. The acquisition parameters and image analysis parameters described above are only representative and are not limited thereto.

  Acquisition parameters may be provided to the cameras 402, 404 and / or the frame buffer 415. For example, the cameras 402, 404 acquire images at the indicated rate in response to acquisition parameters in the operation of the cameras 402, 404, or alternatively, are transferred (per unit time) to the frame buffer 415. The number of acquired frames can be limited. The image analysis parameters can be supplied to the image analysis module 430 as a quantity that affects the operation of the contour definition algorithm.

  The desired values of acquisition parameters and image analysis parameters suitable for a given level of available resources depend on, for example, the characteristics of the image analysis module 430, the type of application that uses the mocap output, and design priorities. obtain. While some image processing algorithms can trade off the approximate contour resolution to the input frame resolution over a wide bandwidth, other image processing algorithms may not offer any great tolerance at all, For example, below it requires a minimum image resolution at which the application will not work together.

  Some implementations are applicable to virtual reality applications or augmented reality applications. For example, referring to FIG. 5, in FIG. 5, an augmented reality experience 213 of a virtual device including a view of a real object, such as a desk surface medium 516 and one or more virtual objects (eg, A system 500 for projecting an object 501) is illustrated. The system 500 includes a processing system 206 that controls various sensors and projectors, such as one or more cameras 102, 104 (or other image sensors) and any number of illumination sources 115, 117, etc., including an imaging system. Prepare. Optionally, a plurality of vibration sensors (or acoustic sensors) 508, 510 that sense contact with the desk 516 may be included. Further, optionally, as a projector controlled by system 206, for example, any audio projector 502 that provides audio feedback, any video projector 504, eg, any haptic projector 506 that provides haptic feedback to an augmented reality user. There is. For further information on projectors, reference is made to “Visio-Tactile Projector” YouTube (https://www.youtube.com/watch?v=Bb0hNMxxewg: searched January 15, 2014). In operation, the sensors and projectors are directed to a target region 212 that can include at least a portion of the desk 516 or a free space in which the target object 214 (in this example, the hand) moves along the indicated path. It is done. One or more applications 521, 522 can be provided as virtual objects integrated within the augmented reality 213 display. Thereby, the user (for example, the owner of the hand 214) can interact with real objects such as the desk 516 and the cola 517 in the same environment as the virtual object 501, for example.

In some implementations, the virtual object is projected to the user. Projection may include an image of an object or other visual display. For example, the visual projection mechanism 504 of FIG. 5 may project a page (eg, virtual device 501) from a book into the reader's augmented reality environment 213 (eg, surface portion 516 and / or surrounding space 212). it can. The result is a virtual device experience of reading a real book or an electronic book on a physical electronic reader, even if no book or electronic reader is present. In some implementations, any haptic projector 506 can project a “virtual paper” property feel of the book onto the reader's finger. In some implementations, an optional audio projector 502 projects a page turning sound upon detecting a reader swipe operation to turn the page.

Claims (17)

A plurality of image sensors arranged to provide stereoscopic image information of the scenery being viewed;
One or more illumination sources arranged around the image sensor;
A controller connected to the image sensor and the illumination source to control operations of the image sensor and the illumination source;
A motion sensing and imaging apparatus characterized in that it acquires landscape image information and provides a user with at least approximately real-time image information pass-through. The controller further provides capturing image information for a control object within a viewable range of the image sensor;
The apparatus of claim 1, wherein the image information for the controlled object is used to determine gesture information that indicates a command to a machine under control. Capturing the image information further comprises:
Separating information received from pixels sensitive to infrared light from information received from pixels sensitive to visible light (eg RGB);
Processing information from an infrared sensor for use in gesture recognition; and
Processing information from the RGB sensor to provide as a live video feed via the presentation interface;
The apparatus according to claim 2, comprising:   Processing the information from the RGB sensor further extracting the entire real-world space features using RGB pixels that separately capture the red, green, and blue elements of the lighting in the landscape. The apparatus of claim 3, comprising:   Processing the information from the infrared sensor further includes extracting fine features of real world space using infrared pixels that capture the infrared elements of the illumination in the scene. The apparatus according to claim 3.   The apparatus according to claim 5, wherein the fine features of the real world space include a surface property of the real world space.   6. The apparatus of claim 5, wherein the fine features of the real world space include an edge of the real world space.   6. The apparatus of claim 5, wherein the subtle features of the real world space include a curvature of the real world space.   The apparatus according to claim 5, wherein the fine features of the real world space include a surface property of an object in the real world space.   6. The apparatus of claim 5, wherein the fine features of the real world space include an edge of an object in the real world space.   6. The apparatus of claim 5, wherein the fine features of the real world space include a curvature of an object in the real world space. The controller further comprises:
Providing ambient lighting conditions and adjusting an output display based on the determined conditions;
The apparatus of claim 1, wherein the first and second position information of the sensor relative to a fixed point is determined at first and second times. A motion sensor,
The controller further comprises:
Determining difference information between the first and second position information from the motion sensor, and providing calculating motion information about the device relative to a fixed point based on the difference information; The apparatus according to claim 1, wherein the apparatus is characterized.   The apparatus according to claim 1, further comprising one or more fixing members that attach the image sensor and the illumination source to an attachment surface of the wearable presentation apparatus.   The apparatus according to claim 1, further comprising one or more fixing members that attach the image sensor and the illumination source to a hollow portion of the wearable presentation apparatus. The apparatus according to claim 1, further comprising at least one fixing member that attaches the image sensor and the illumination source to a mounting surface of the portable presentation device.
The apparatus according to claim 1. The apparatus according to claim 1, further comprising at least one fixing member that attaches the image sensor and the illumination source to a hollow portion of the portable presentation device.