WO2025057379A1 - Posture estimation system, posture estimation method, and program - Google Patents

Abstract

According to the present invention, a posture estimation system for estimating a posture using key points more appropriately acquires information indicating a portion constituting an object and hidden by a hand (S204, S402), determines three-dimensional positions of a plurality of key points for estimating the posture of the object on the basis of the information (S208, S405), trains a machine learning model for estimating the positions of a plurality of key points determined in an input image (S203, S407), acquires the estimated positions of key points in an image capturing the object and the hand on the basis of an output produced by the trained machine learning model in response to receiving the image, and determines the estimated posture of the object in a three-dimensional space on the basis of the estimated positions of the key points.

Description

Posture estimation system, posture estimation method, and program

The present invention relates to a posture estimation system, a posture estimation method, and a program.

There is a technique for estimating the positions of an object's keypoints from an image of the object, and then estimating the object's pose from the estimated keypoints. The three-dimensional positions of the object's keypoints are determined in advance. For example, a machine learning model is trained to predict the positions of keypoints in an image, and the machine learning model is used to estimate the positions of keypoints in an image from a captured image.

Sida Peng et al. published a paper entitled PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation at the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). This paper discloses that a machine learning model is trained using training data including input images generated from a 3D model and correct output images, and that the positions of key points in the image used for pose estimation are calculated based on the output when a captured image is input to the machine learning model.

When estimating pose, there are cases where it is difficult to estimate the position of key points from an image, for example when an object is hidden by a hand. This can lead to a decrease in the accuracy of pose estimation or a decrease in processing speed.

The present invention was made in consideration of the above situation, and its purpose is to provide a technology that enables posture estimation to be performed more appropriately.

In order to solve the above problems, the posture estimation system of the present invention includes one or more processors, which acquire information indicating parts of an object that are hidden by a hand, determine the three-dimensional positions of a number of key points for estimating the posture of the object based on the acquired information, train a machine learning model for estimating the positions of the determined number of key points in an input image, acquire positions of estimated key points in the image based on the output when an image of an object and a hand is input to the trained machine learning model, and determine an estimated posture of the object in three-dimensional space based on the positions of the estimated key points.

In one form of the invention, the information indicating the portion obscured by the hand is a plurality of images in which the object is held by the hand, and the one or more processors may determine three-dimensional positions of the plurality of keypoint candidates determined by a predetermined procedure based on the frequency with which the hand obscures the plurality of keypoint candidates in the plurality of images in which the object is held by the hand.

In one embodiment of the present invention, the information indicating the portion hidden by the hand may be the portion of the object indicated by the user that is being held by the hand.

In one embodiment of the present invention, the information indicating the portion hidden by the hand is information indicating the portion of the object designated by the user and associated with a tag, and the one or more processors may determine whether the portion associated with the tag is being operated by the hand based on an image of the object and hand, and execute processing according to the tag if it is determined that the portion is being operated.

In one embodiment of the present invention, the one or more processors may determine whether a part associated with the tag is being operated by the hand based on an image of the object and hand, and if it is determined that the part is being operated, may execute processing corresponding to the tag based on the magnitude of the operation by the hand.

In addition, the posture estimation method of the present invention involves using one or more processors to acquire information indicating parts of an object that are hidden by the hand, determining the three-dimensional positions of a number of key points for estimating the posture of the object based on the acquired information, acquiring a trained machine learning model for estimating the positions of the determined number of key points in an input image, acquiring the positions of the estimated key points in the image based on the output when an image of an object and a hand is input to the acquired machine learning model, and estimating the posture of the object in three-dimensional space based on the positions of the estimated key points.

The program of the present invention causes a computer to function as: an acquisition means for acquiring information indicating parts of an object that are hidden by a hand; a key point determination means for determining three-dimensional positions of a number of key points for estimating the posture of the object based on the acquired information; a model acquisition means for acquiring a trained machine learning model for estimating the positions of the determined number of key points in an input image; a position acquisition means for acquiring positions of the estimated key points in the image based on an output when an image of an object and a hand is input to the acquired machine learning model; and a posture estimation means for estimating the posture of the object in three-dimensional space based on the positions of the estimated key points.

The present invention makes it possible to more appropriately estimate posture using keypoints.

1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. 2 is a block diagram showing an example of functions implemented in an information processing system according to an embodiment of the present invention; FIG. FIG. 2 is a diagram showing an example of a photographed image of an object. 10A and 10B are diagrams illustrating an example of a tag area associated with a function tag. FIG. 2 is a flow chart illustrating an outline of processing of the information processing system. FIG. 11 is a flow diagram illustrating an example of a process for determining keypoints and training an estimation model. FIG. 10 is a diagram illustrating an example of key point candidates generated from an object. FIG. 11 is a flow diagram showing an example of a process for generating training data and learning an estimation model. FIG. 11 is a diagram illustrating an example of correct answer data. FIG. 11 is a flow diagram showing another example of a process for determining keypoints and training an estimation model.

Below, one embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, a case will be described in which the invention is applied to an information processing system that inputs an image of an object, estimates its posture, and draws an image according to the estimated posture.

This information processing system includes a machine learning model that outputs information indicating the estimated pose of an object from an image in which the object is captured.

FIG. 1 is a diagram showing an example of the configuration of an information processing system according to one embodiment of the present invention. The information processing system according to this embodiment includes an information processing device 10. The information processing device 10 is, for example, a computer such as a game console, a personal computer, or a VR headset. As shown in FIG. 1, the information processing device 10 includes, for example, a processor 11, a memory unit 12, a communication unit 13, an operation unit 16, a display unit 18, and an imaging unit 20. The information processing system may be composed of one information processing device 10, or may be composed of multiple devices including the information processing device 10, and for example, the imaging unit 20 or the display unit 18 may be located in a housing separate from the information processing device 10.

The processor 11 is, for example, a program-controlled device such as a CPU that operates according to a program installed in the information processing device 10.

The storage unit 12 is composed of at least a portion of a memory element such as a ROM or RAM, or an external storage device such as a solid-state drive. The storage unit 12 stores programs executed by the processor 11, etc.

The communication unit 13 is a communication interface for wired or wireless communication, such as a network interface card, and transmits and receives data between other computers and terminals via a computer network such as the Internet.

The operation unit 16 is an input device such as a keyboard, mouse, touch panel, or game console controller, and receives operation input from the user and outputs a signal indicating the content of the input to the processor 11.

The display unit 18 is a display device such as a liquid crystal display, and displays various images according to instructions from the processor 11. The display unit 18 may be built into the VR headset, or may be a device that outputs a video signal to an external display device.

The imaging unit 20 is a photographing device including an image sensor. The imaging unit 20 may be a camera capable of acquiring visible RGB images. The imaging unit 20 may be a camera capable of acquiring visible RGB images and depth information synchronized with the RGB images. The imaging unit 20 in this embodiment may be, for example, a camera capable of capturing moving images, or may be a camera built into a VR headset. The imaging unit 20 may be outside the information processing device 10, in which case the information processing device 10 and the imaging unit 20 may be connected via the communication unit 13 or an input/output unit described below.

The information processing device 10 may also include audio input/output devices such as a microphone and a speaker. The information processing device 10 may also include, for example, a communication interface such as a network board, an optical disk drive that reads optical disks such as DVD-ROMs and Blu-ray (registered trademark) disks, and an input/output unit (USB (Universal Serial Bus) port) for inputting and outputting data to and from external devices.

FIG. 2 is a block diagram showing an example of functions implemented in an information processing system according to one embodiment of the present invention. As shown in FIG. 2, the information processing system functionally includes a posture estimation unit 25, a tag processing unit 29, an image rendering unit 30, a shape model acquisition unit 31, an occlusion information acquisition unit 32, and a learning control unit 35. The posture estimation unit 25 functionally includes an estimation model 26, a position acquisition unit 27, and a posture determination unit 28. The learning control unit 35 functionally includes a key point determination unit 36 and an estimation learning unit 37. The estimation model 26 is a type of machine learning model.

These functions are mainly implemented by the processor 11 and the storage unit 12. More specifically, these functions may be implemented by having the processor 11 execute a program that is installed in the information processing device 10, which is a computer, and that includes execution instructions corresponding to the above functions. In addition, this program may be supplied to the information processing device 10 via, for example, a computer-readable information storage medium such as an optical disk, a magnetic disk, or a flash memory, or via the Internet, etc.

Note that the information processing system according to this embodiment does not necessarily have to implement all of the functions shown in FIG. 2, and may also implement functions other than those shown in FIG. 2.

The posture estimation unit 25 estimates the posture of the target object based on the information output when an input image is input to the estimation model 26. The input image is an image of the object captured by the imaging unit 20. The estimation model 26 is a machine learning model that is trained using training data, and when input data is input, the trained estimation model 26 outputs data as an estimation result.

FIG. 3 is a diagram showing an example of an image of a photographed object. The target object 51 shown in FIG. 3 is held, for example, by a hand 53, and is photographed by the photographing unit 20.

Information on an image of a target object is input to the trained estimation model 26, and the estimation model 26 outputs information indicating the positions of keypoints for estimating the posture of the object. More specifically, the estimation model 26 outputs an image indicating the positions of each of a number of keypoints set for the object. An estimation model 26 may exist for each keypoint or each keypoint candidate.

The training data for the estimation model 26 includes multiple learning images rendered by a three-dimensional shape model of the target object, and ground truth data indicating the positions of the object's keypoints in the learning images. Keypoints are virtual points within an object that are used to calculate the pose. The data output by the estimation model 26 may be a position image in which each point indicates the positional relationship between that point and a keypoint (e.g., relative direction), or a position image that is a heat map in which each point indicates the probability that a keypoint exists. The learning of the estimation model 26 will be described in detail later.

The input image may be an image that has been processed from an image of an object captured by the image capture unit 20. For example, it may be an image in which the area excluding the target object is masked, or an image that has been enlarged or reduced so that the size of the object in the image is a predetermined size.

The position acquisition unit 27 determines the two-dimensional position of the keypoint in the input image based on the output of the estimation model 26 when an image of an object and a hand is input to the trained estimation model 26. For example, the position acquisition unit 27 determines candidates for the two-dimensional position of the keypoint in the input image based on the position image output from the estimation model 26. For example, the position acquisition unit 27 calculates candidate points for the keypoint from each combination of any two points in the position image, and generates a score indicating whether the multiple candidate points match the direction indicated by each point in the position image. The position acquisition unit 27 may estimate the candidate point with the largest score as the position of the keypoint. The position acquisition unit 27 also repeats the above process for each keypoint.

The pose determination unit 28 estimates the pose of the object based on information indicating the two-dimensional positions of keypoints in the input image and information indicating the three-dimensional positions of keypoints in a three-dimensional shape model of the target object, and outputs pose data indicating the estimated pose. The pose of the object is estimated using a known algorithm. For example, it may be estimated using a solution to the Perspective-n-Point (PNP) problem for pose estimation (e.g., EPnP). Furthermore, the pose determination unit 28 may estimate not only the pose of the object but also the position of the object in the input image, and the pose data may include information indicating that position.

The internal camera parameters of the image capture unit 20 are assumed to have been acquired in advance through calibration. These parameters are used when solving the PnP problem.

Details of the estimation model 26, the position acquisition unit 27, and the attitude determination unit 28 may be those described in the paper PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation.

The tag processing unit 29 determines whether the part associated with the functional tag is being operated by the hand, based on information indicating the part of the object to which the functional tag is associated, and an image of the object and the hand. If it is determined that the part is being operated, the tag processing unit 29 executes processing according to the functional tag. If it is determined that the part is being operated, the tag processing unit 29 may execute processing according to the tag based on the size of the hand operation.

FIG. 4 is a diagram illustrating an example of tag regions 61, 62 associated with functional tags. Each of the tag regions 61, 62 is a part of the target object 51. The tag regions 61, 62 may be regions on the surface of the target object 51, or may be three-dimensional regions including the interior. The tag regions 61, 62 are associated with different functional tags. For example, the tag region 61 may be associated with the functional tag of a switch, and the tag region 62 may be associated with the functional tag of a part to be grasped. The tag regions 61, 62 may be any of a region that can be grasped by the user, a region that displays a virtual touch screen and allows touch interaction, and a region that emits a light source or particles for lighting purposes, and each may be associated with a functional tag corresponding to that region.

The image rendering unit 30 renders an image based on the estimated orientation of the object. The image rendering unit 30 may render a three-dimensional image of the object based on the estimated orientation of the object and a three-dimensional shape model. The image rendering unit 30 may determine the orientation of an object to be rendered, such as an object in a VR image, based on the estimated orientation of the object, and render the object to be rendered.

The shape model acquisition unit 31 acquires multiple captured images of a target object captured by the imaging unit 20. The shape model acquisition unit 31 generates and acquires a three-dimensional shape model of the object from the multiple captured images. More specifically, the shape model acquisition unit 31 extracts multiple feature vectors indicating local features for each of the multiple captured images, and determines the three-dimensional position of the point from which the feature vector was extracted from the multiple corresponding feature vectors extracted from the multiple captured images and the position from which the feature vector was extracted in the captured image. The shape model acquisition unit 31 then acquires a three-dimensional shape model of the object based on the three-dimensional position. This method is a well-known method that is also used in software that realizes so-called SfM and Visual SLAM, so a detailed explanation will be omitted.

The occlusion information acquisition unit 32 acquires information indicating the parts of the target object that are hidden by the hand. At this time, it is assumed that the hand is holding the object. More specifically, the information indicating the parts hidden by the hand is at least a part of a plurality of images in which the target object is held by the hand, and information indicating the parts of the target object that are specified by the user as the parts that are held by the hand.

The occlusion information acquisition unit 32 may acquire multiple images of the target object being held by the hand, captured by the image capture unit 20, as information indicating the portion obscured by the hand.

The occlusion information acquisition unit 32 may acquire information indicating a part of an object designated by a user and held by the hand as information indicating a part hidden by the hand. Tag regions 61, 62 may be designated as parts of the object.

The occlusion information acquisition unit 32 may input information about the target object into a trained machine learning model that estimates the area held by the hand, and identify parts of the object using the output of the machine learning model. This machine learning model is publicly known, so a detailed description of it will be omitted.

The learning control unit 35 determines the key points of the target object based on the three-dimensional shape model of the object and trains the estimation model 26.

The key point determination unit 36 may determine the three-dimensional positions of multiple key points for estimating the posture of the target object based on a three-dimensional shape model of the target object and information indicating the parts hidden by the hands. If the information indicating the parts hidden by the hands is multiple images of the object being held by the hands, the key point determination unit 36 may determine multiple key points based on the frequency with which multiple key point candidates determined by a predetermined method are hidden by the hands in the multiple images, and may determine the three-dimensional positions of the determined key points.

The keypoint determination unit 36 may generate a set of multiple keypoint candidates, for example, by using the well-known Farthest Point algorithm. The number of keypoints N may be an integer greater than or equal to 4. The number of keypoint candidates may be an integer greater than the number of keypoints N (for example, greater than or equal to 1.3 times the number of keypoints).

If the information indicating the part hidden by the hand is information indicating the part of the object indicated by the user and held by the hand, the key point determination unit 36 may determine multiple key points from multiple key point candidates based on that part, and may determine the three-dimensional positions of the determined key points.

The keypoint determination unit 36 may determine multiple keypoints from multiple keypoint candidates based on the reliability of the posture estimation using the keypoints. The method of calculating the reliability will be described later.

The estimation learning unit 37 trains the estimation model 26, which is a machine learning model for estimating the positions of the determined multiple key points in the input image. More specifically, the estimation learning unit 37 generates training data used to train the estimation model 26, and trains the estimation model 26 using the training data.

The training data includes a number of training images rendered using a three-dimensional shape model of the target object, and ground truth data indicating the positions of the object's keypoints in the training images. At least in the initial training data, the keypoints for which the estimation learning unit 37 generates ground truth data may be those included in a set of keypoint candidates. The estimation learning unit 37 may generate ground truth data for all keypoint candidates included in the initial set, and train the estimation model 26.

More specifically, the estimation learning unit 37 may determine the positions of keypoint candidates in the learning images based on the pose of the rendered object, and generate a correct position image for each of the keypoint candidates according to its position. The training data may include learning images in which the object is photographed, and position images generated from the pose of the object in the learning images estimated by so-called SfM or Visual SLAM.

In this embodiment, the estimation learning unit 37 trains an estimation model 26 for each of the keypoint candidates. The estimation model 26 for the selected keypoint candidate is used as the keypoint estimation model 26 for pose estimation (inference processing) for the input image.

The processing of the information processing system is explained below. Figure 5 is a flow diagram that shows an overview of the processing of the information processing system.

First, the information processing system generates a three-dimensional shape model of a target object using a known method based on an image of the object (S101).

Then, the learning control unit 35 included in the information processing system determines the positions of the key points based on the three-dimensional shape model and information indicating the parts hidden by the hand, and trains the estimation model 26 for pose estimation (S102).

Once the estimation model 26 has been trained, the posture estimation unit 25 inputs an input image of an object into the trained estimation model 26 (S103) and obtains data output by the estimation model 26. Then, based on the output of the estimation model 26, the two-dimensional positions of key points in the image are determined (S104).

More specifically, if the output of the estimation model 26 is a position image in which each point indicates the relative direction to a keypoint, the position acquisition unit 27 included in the posture estimation unit 25 calculates candidates for the position of the keypoint from each point of the position image, and determines the position of the keypoint based on the candidates. If the output of the estimation model 26 is a position image of a heat map, the position acquisition unit 27 determines the position of the most probable point as the position of the keypoint using a known method.

The posture estimation unit 25 estimates the posture of the object based on the two-dimensional positions of the determined keypoints and the three-dimensional positions of those keypoints in the three-dimensional shape model (S105).

The tag processing unit 29 also acquires information indicating the hand pose from the input image (S106). As the hand pose, an input image of a photographed finger or coordinates of joint points in three-dimensional space may be acquired. In acquiring the hand pose, a machine learning model trained using an image and ground truth data indicating the joint points may be used. The input image may include not only a visible image but also a depth image. Techniques for acquiring the hand pose are well known, so detailed explanations are omitted.

The tag processing unit 29 determines whether the hand is touching a part of the object corresponding to the function tag based on the acquired information indicating the hand pose (S107). The tag processing unit 29 may determine whether the hand is touching based on whether the distance between the three-dimensional coordinates of any joint point of the hand and a part associated with the function tag (e.g., tag areas 61, 62) is equal to or less than a threshold.

If it is determined that the hand is touching the part corresponding to the function tag (S107), the tag processing unit 29 executes processing according to the function tag corresponding to that part (S108). On the other hand, if it is determined that the hand is not touching the part corresponding to the function tag, the processing of S108 is skipped.

Then, the image drawing unit 30 draws an image based on the estimated posture (S108) and displays the drawn image on the display unit 18. The image may also be displayed on another display.

In the example of FIG. 5, the process from S103 to S109 is described as being performed once, but in reality, the process from S103 to S109 may be executed repeatedly, and the posture may be estimated and the image may be drawn in real time in response to the movement of the object.

FIG. 6 is a flow diagram showing an example of the process of determining key points and learning the estimation model 26. FIG. 6 is a diagram explaining the process of S102 in FIG. 3 in more detail.

First, the key point determination unit 36 generates multiple key point candidates (S201). More specifically, the key point determination unit 36 may generate multiple key point candidates and their three-dimensional positions from a three-dimensional shape model of the object (more specifically, information on vertices included in the three-dimensional shape model), for example, by using the well-known Farthest Point algorithm.

FIG. 7 is a diagram illustrating an example of keypoint candidates generated from an object. FIG. 7 shows an example of keypoints generated when a different object from those in FIGS. 3 and 4 is targeted. For ease of explanation, seven keypoint candidates K1 to K7 are shown in FIG. 7, but more keypoint candidates may be generated.

Once the keypoint candidates are generated, the estimation learning unit 37 generates training data for the estimation model 26 (S202). The training data includes a training image rendered based on the three-dimensional shape model and ground truth data indicating the positions of each of the keypoint candidates in the training image.

FIG. 8 is a flow diagram showing an example of a process for generating training data. FIG. 8 is a diagram explaining the process of S202 in more detail. First, the estimation learning unit 37 acquires data of a three-dimensional shape model of an object (S301). Then, the estimation learning unit 37 acquires multiple viewpoints for rendering (S302). More precisely, the estimation learning unit 37 acquires multiple camera viewpoints for rendering and shooting directions corresponding to the camera viewpoints. The multiple camera viewpoints may be provided at positions at a constant distance from the origin of the three-dimensional shape model, and the shooting direction is a direction from the camera viewpoints toward the origin of the three-dimensional shape model.

Once the viewpoints are acquired, the estimation learning unit 37 renders an image of the object for each viewpoint based on the three-dimensional shape model (S303). The images may be rendered using a known method.

Once the image is rendered, the estimation learning unit 37 adds the rendered image as a training image to the training data together with the viewpoint (S304). Here, the estimation learning unit 37 may perform a predetermined data augmentation on the rendered image and use the converted image as the training image. In the data augmentation method, for example, the rendered image may be transformed by disturbing at least a portion of the luminance, saturation, and hue of the image, or by cropping out a portion of the image and resizing it to the same size as the original.

The estimation learning unit 37 may further add a photographed image of the object with a viewpoint to the training image. This photographed image may be the photographed image used to generate the three-dimensional shape model. The camera viewpoint of the photographed image may be the camera viewpoint acquired when generating the three-dimensional shape model.

Once the training images are prepared, the estimation learning unit 37 generates correct answer data indicating the positions of the keypoints in each training image based on the three-dimensional positions of the keypoint candidates and the viewpoint of the training image for each training image (S305). The estimation learning unit 37 generates correct answer data for each keypoint candidate for each training image.

FIG. 9 is a diagram showing an example of the correct answer data. The correct answer data is information indicating the two-dimensional positions of key points of an object in a training image, and may be a position image in which each point indicates the positional relationship (e.g., direction) between that point and the key point.

A position image may be generated for each type of keypoint. The position image indicates the relative direction of each point between that point and the keypoint. In the position image shown in Figure 9, a pattern is depicted according to the value of each point, and the value of each point indicates the direction between the coordinates of that point and the coordinates of the keypoint. Figure 9 is merely a schematic diagram, and the actual value of each point changes continuously. The position image in Figure 9 is a Vector Field image that indicates the relative direction of the keypoint with respect to that point.

The process shown in Figure 8 generates training data that includes training images and correct answer data.

Once the training data is generated, the estimation learning unit 37 uses the training data to train an estimation model 26 for each keypoint candidate (S203). The trained estimation model 26 is used to detect parts of an object that are hidden by a hand, for example, by the method described below.

Once the estimation model 26 has been trained, the key point determination unit 36 outputs an instruction to the user to move the object being held in the user's hand in front of the image capture unit 20. The user follows the instruction to move the object being held in front of the image capture unit 20.

Then, the key point determination unit 36 acquires an image of the object held in the hand captured by the image capture unit 20, and further acquires the posture of the object in the image (S204). The key point determination unit 36 may acquire an image of the object that constitutes a video. In acquiring the posture of the object, the key point determination unit 36 may determine the two-dimensional position of a key point candidate based on information output when the image is input to the trained estimation model 26, and acquire the posture by processing similar to that of the posture determination unit 28 based on the two-dimensional position of the key point candidate and its position in the three-dimensional shape model. Note that if the difference between the acquired posture and the posture acquired from the previous image is equal to or less than a threshold value, or if the image of the object and the previously captured image are similar, the key point determination unit 36 may discard the image and repeat the processing of S204.

If the posture cannot be acquired due to reasons such as failure to estimate key points, the key point determination unit 36 may acquire an image and posture of the object by having the user adjust the object so that it assumes a specified posture. The key point determination unit 36 may display a rendering image of the specified object on a VR headset or the like, and adjust the position and posture of the object being held so that it overlaps with the rendering image.

Once the images are acquired, the key point determination unit 36 extracts hand regions from each of the images (S205). The hand regions may be extracted simply based on color, or may be extracted using a publicly known trained machine learning model.

The key point determination unit 36 determines whether each of the key point candidates is occluded in the extracted hand region (S206). The key point determination unit 36 may determine that a key point candidate is occluded when the position of the key point candidate in the image is within the extracted hand region.

Then, the key point determination unit 36 checks whether a repetition end condition is met (S208). The repetition end condition may be that the number of images that have been subject to judgment so far is equal to or greater than a threshold value, or that when the surface of a virtual sphere surrounding the object is divided into multiple parts, all parts are associated with the posture. The part that is in the direction indicated by the posture obtained from the image may be the part associated with the posture.

If the repetition end condition is not met (N in S207), the processes from S204 onwards are repeated. On the other hand, if the repetition end condition is met (Y in S207), the keypoint determination unit 36 determines keypoints based on the frequency with which each of the keypoint candidates is determined to be occluded and the reliability of the pose estimation (S208).

More specifically, the keypoint determination unit 36 selects a provisional set of keypoints from the multiple keypoint candidates based on the frequency with which each of the keypoint candidates is determined to be occluded. As the initial provisional set, a predetermined number of keypoints that are less frequently occluded among the keypoint candidates may be selected. The keypoint determination unit 36 obtains the reliability of the pose estimation for the keypoints when the pose estimation unit 25 estimates the pose for the image acquired in S204.

The reliability may be determined based on the pose of the object estimated by the pose determination unit 28 and the correct pose. For example, the key point determination unit 36 may calculate the pose obtained from the image using SLAM technology or the like as the correct answer, and calculate the reliability based on the difference between the correct pose and the estimated pose.

The keypoint determination unit 36 may also reproject the position of each of the keypoint candidates in the image based on the pose estimated by the tentative keypoints and the three-dimensional positions of the keypoint candidates, and store the reprojected positions in the storage unit 12. In this case, the keypoint determination unit 36 may calculate, as the reliability, the average of the distances between the position estimated by the output of the estimation model 26 and the reprojected position for each of the keypoint candidates.

If the reliability is higher than a threshold, the keypoint determination unit 36 determines the keypoints in the set as official keypoints; if not, it selects a provisional set consisting of keypoints different from the previous set from the multiple keypoint candidates, and repeats the process from obtaining the reliability onwards. The new provisional set may be generated, for example, by replacing a randomly selected keypoint from the original set with one of the unselected keypoint candidates. The keypoint candidate to be replaced may be determined, for example, based on a score calculated based on low frequency and distance from the keypoints in the existing set.

Once the set of key points is determined, the learning control unit 35 sets the posture estimation unit 25 to estimate the posture using the determined set of key points (S209). Note that the estimation model 26 used for actual posture estimation may be one that has already been trained on the key points (key point candidates).

The processing from S204 to S208 makes it possible to efficiently eliminate keypoints that are likely to be obscured by hands and that are likely to have a negative effect on pose estimation accuracy, making it possible to perform pose estimation more efficiently. In addition, by using the reliability of pose estimation, it is possible to prevent a decrease in pose estimation accuracy, for example, due to keypoints concentrating in a small area. In addition, because keypoint candidates that have a low contribution to pose estimation can be eliminated in advance, it is possible to achieve both accuracy and processing speed in pose estimation, making processing more efficient.

The keypoint candidates that are the subject of the processing in S208 may be those whose frequency of obscuration is equal to or less than a threshold. Here, if the number of keypoint candidates is less than the number of keypoints plus a predetermined number (e.g., 2), the keypoint determination unit 36 may generate additional keypoint candidates to replace the keypoint candidates whose frequency of obscuration exceeds the threshold, and re-execute the processing from S202 onwards for the replaced keypoint candidates.

In the process shown in FIG. 6, the part that is actually hidden by the hand is obtained, but instead, information indicating the part of the object specified by the user that is being held by the hand may be obtained as information indicating the part that is hidden by the hand.

FIG. 10 is a flow diagram showing another example of the process of determining key points and learning the estimation model 26. In this example, the user manually specifies, as a tag region, a portion of an image of an object shown on a display that is obscured by a hand.

First, the key point determination unit 36 displays an image of an object based on a three-dimensional shape model (S401). Next, based on the user's operation on the image, it acquires a tag area of the object designated by the user and a function tag designated for the tag area (S402).

The key point determination unit 36 may display an icon of a paint tool including a paint palette along with an image of the object, and execute a process of coloring the scanned model with a color specified by the user using the paint palette. The user may paint any position while holding the object. In this case, an image of a virtual object of the same shape may be transparently superimposed on the image of the actual object, and the virtual object may be colored to visualize the area as if it were painted on the real thing. The paint color may also be associated with a functional tag.

The key point determination unit 36 may acquire this colored area as a tag area. Alternatively, instead of coloring with a paint tool, the key point determination unit 36 may specify the tag area by having the user select one of a plurality of virtual stickers that each correspond to a functional tag and attach the selected virtual sticker to the object.

The keypoint determination unit 36 generates multiple keypoint candidates (S403) in parallel with S401 and S402. This process is similar to S201, so a detailed explanation will be omitted.

The key point determination unit 36 calculates the probability that each of the key point candidates is hidden in a tag area that satisfies a predetermined condition (S404). The tag area that satisfies the predetermined condition may be, for example, a tag area designated as an area to be grasped, or a tag area associated with a function that is touched by the hand, such as a switch.

The keypoint determination unit 36 may, for example, cast rays in multiple (isotropic) directions from the keypoint candidate, and further calculate a value indicating the ratio of rays that hit the tag region as the probability of that keypoint candidate. Furthermore, if the tag region is a three-dimensional region, the keypoint determination unit 36 may set the probability value to 1 if the keypoint candidate is within that region, and may set the probability value to 0 if it is not.

The keypoint determination unit 36 selects keypoints to be used for final pose estimation based on the probability that the keypoint candidates are occluded (S405). Here, the keypoint determination unit 36 may select a predetermined number of keypoints from the keypoint candidates with low probability.

Alternatively, the keypoint determination unit 36 may select a keypoint based on the reliability and the probability. For example, the keypoint determination unit 36 may determine a tentative keypoint and calculate the reliability based on the tentative keypoint. The keypoint determination unit 36 generates a score indicating the suitability of the keypoint as a keypoint from the reliability and the probability of each keypoint, and determines the keypoint based on the score. The reliability may be calculated in the following manner. First, the estimation model 26 learned about the tentative keypoint is used to estimate the posture of the image captured when the posture estimation unit 25 generates the three-dimensional shape model. Next, the keypoint determination unit 36 reprojects the positions of each of the keypoint candidates in the image based on the estimated posture and the three-dimensional positions of the keypoint candidates. Then, the keypoint determination unit 36 calculates the average of the distance between the position estimated by the output of the estimation model 26 and the reprojected position as the reliability for each of the keypoint candidates. Note that the keypoints may be determined iteratively using a method similar to S208.

Once the keypoints are determined, the estimation learning unit 37 generates training data for learning the estimation model 26 for each of the keypoint candidates (S406). The estimation learning unit 37 also trains the estimation model 26 for each of the keypoints (S407). Note that if the keypoints (keypoint candidates) have been learned in advance, there is no need to redundantly train the estimation model 26.

The method shown in Figure 10 also makes it possible to efficiently eliminate keypoints that are likely to be hidden by the hand and thus have a high probability of adversely affecting pose estimation when the object is held in the hand, making it possible to perform pose estimation more efficiently. In addition, because it is possible to eliminate keypoint candidates that have a low contribution to pose estimation in advance, it is possible to achieve both accuracy and processing speed in pose estimation, making processing more efficient.

In addition, in the method shown in FIG. 10, the part hidden by the hand is identified by assigning some kind of function. Therefore, the operation for identifying the part hidden by the hand itself can be omitted, improving convenience.

Note that the specific numerical values above and the objects and numerical values in the drawings are merely examples, and are not limited to these examples and may be modified as necessary.

Claims (7)

one or more processors;
The one or more processors:
acquiring information indicating a portion of an object that is hidden by a hand;
determining three-dimensional positions of a number of key points for estimating a pose of the object based on the acquired information;
training a machine learning model to estimate locations of the determined keypoints in an input image;
Obtaining estimated positions of key points in an image based on an output of the trained machine learning model when the image includes an object and a hand;
determining an estimated pose of the object in three-dimensional space based on the estimated keypoint locations;
Pose estimation system. 2. The posture estimation system according to claim 1,
the information indicating the portion obscured by the hand is a plurality of images of the object being grasped by the hand;
the one or more processors determine three-dimensional positions of the plurality of keypoints based on a frequency with which the plurality of keypoint candidates determined by a predetermined procedure are occluded by the hand in the plurality of images in which the object is held by the hand;
Pose estimation system. 2. The posture estimation system according to claim 1,
The information indicating the portion hidden by the hand is a portion of the object indicated by the user and grasped by the hand.
Pose estimation system. 4. The posture estimation system according to claim 3,
the information indicating the portion hidden by the hand is information indicating a portion of the object designated by a user and to which a tag is associated;
the one or more processors determine whether a part associated with the tag is being operated by the hand based on an image of the object and the hand, and execute a process corresponding to the tag when it is determined that the part is being operated.
Pose estimation system. 5. The posture estimation system according to claim 4,
the one or more processors determine whether a part associated with the tag is being operated by the hand based on an image of the object and the hand, and when it is determined that the part is being operated, executes a process corresponding to the tag based on a size of the operation by the hand.
Pose estimation system. by one or more processors,
acquiring information indicating a portion of an object that is hidden by a hand;
determining three-dimensional positions of a number of key points for estimating a pose of the object based on the acquired information;
obtaining a trained machine learning model for estimating locations of the determined keypoints in an input image;
Obtaining estimated positions of key points in the image based on an output of the obtained machine learning model when an image of an object and a hand is input;
estimating a pose of the object in a three-dimensional space based on the estimated keypoint positions;
Pose estimation methods. an acquisition means for acquiring information indicating a part of an object that is hidden by a hand;
a key point determining means for determining three-dimensional positions of a plurality of key points for estimating a pose of the object based on the acquired information;
a model acquisition means for acquiring a trained machine learning model for estimating positions of the determined plurality of keypoints in an input image;
A position acquisition means for acquiring positions of estimated key points in the image based on an output when an image of an object and a hand is input to the acquired machine learning model; and
a posture estimation means for estimating a posture of the object in a three-dimensional space based on the estimated positions of the key points;
A program that makes a computer function as a

Images