CN114419716A - A calibration method for face key point calibration in face images - Google Patents

Abstract

The present invention provides a calibration method for calibrating facial key points for continuous face images, including: using the key points of the first face image obtained through the DAN network as candidate key points, Each key point of the image is calibrated in turn: obtain the regional gradient feature vector, calculate the similarity result α ₁ of the regional gradient feature vector before and after the frame; obtain the regional convolution feature, calculate the similarity of the regional convolution feature in the face image before and after the frame Result α ₂ ; the similarity result α ₁ and the similarity result α ₂ are added according to the proportion to obtain the similarity result β: if it is greater than the preset threshold, the candidate key point is the key point after the calibration of the face image of the next frame ; If it is less than the preset threshold, search in the preset key point search area until the key points greater than the threshold are obtained as the calibrated key points of the next frame of face image. The method improves the positioning accuracy of face key points and the anti-noise effect, so that it can be applied to fields that require higher accuracy of face key points.

Description

A calibration method for face key point calibration in face images

technical field

The invention relates to the technical field of machine vision, in particular to a calibration method for calibrating facial key points in a human face image.

Background technique

In recent years, there have been more and more researches on face analysis. The so-called face analysis refers to the recognition of human expressions and identities, and the accurate identification of face key points through computer vision-related technologies based on face images. Localization is an important basic link in face analysis tasks. Face key point location is usually the preprocessing work for all face analysis tasks, and face alignment is required for data standardization in the follow-up.

The methods of face keypoint detection can be divided into two categories: detection methods based on point distribution (PDM) model and detection methods based on deep learning.

Among them, the detection method based on the point distribution model mainly includes the following parts: performing statistical analysis on the face key point samples, concatenating the coordinates of all key points to obtain a vector representation, and obtaining a model reflecting the distribution law of face key points through the least squares method - Point distribution model. Traditional algorithms such as ASM and AAM will form a shape model based on statistics, which is highly interpretable, but its shape model and detection results are strongly dependent on the data set, and the generalization performance is poor. At the same time, because its statistics are based on the overall average , which results in its poor performance in the face of some special data. The detection method based on deep learning takes the entire image as the network input, which fully avoids the sparsity of features and can learn more information; at the same time, the deep convolutional neural network can learn deep semantic features. Although the emergence of deep learning technology has improved the performance of the face key point algorithm to a certain extent, there are still some shortcomings, such as face pose and occlusion. Although there have been some methods to deal with related problems in recent years, but in low-quality images In real-time situations, the current method is still far from practical application.

In the prior art, the method for locating key points of a face mainly includes using a geometric relationship to obtain an estimated position after decomposing and symmetric transforming the image containing the face. However, in many difficult scenes, the results of facial key point location are still unsatisfactory, and there are many external factors that affect the detection accuracy, including pose, occlusion, expression, and lighting. In an unconstrained environment, the task of facial key point localization is difficult due to the changes caused by the facial features themselves or the environment. The traditional face key points can achieve better results when dealing with tasks that require less precision, such as face recognition, but face micro-expression recognition, face pose recognition and other tasks that require higher precision. big.

In order to solve these problems, the present invention proposes a new calibration method for the calibration of facial key points in human face images.

SUMMARY OF THE INVENTION

In order to solve the above problems, the present invention provides a calibration method for calibrating facial key points in a face image.

The present invention provides the following technical solutions.

A calibration method for calibrating facial key points in a face image, comprising the following steps:

Based on the coordinates of the key points of the first face image obtained by the trained DAN network, the key points of the face image of the next frame are calibrated in turn, including the following steps:

Taking the calibrated key point coordinates in the face image of the previous frame as the coordinates of the candidate key points of the subsequent frame, respectively, establish the gradient feature vector generation area of the two frames of images as the center;

By calculating the gradient magnitude and gradient direction of the gradient feature vector generation region, the regional gradient feature vector is obtained, and the similarity result α ₁ of the regional gradient feature vectors of the two frames of face images before and after is calculated;

The convolution feature comparison area of the two frames of images is established with the key points as the center, and the convolution feature comparison area is convolved multiple times to obtain the regional convolution features, and the similarity of the regional convolution features in the two frames of face images before and after is calculated. Sexual result α ₂ ;

The similarity result α ₁ and the similarity result α ₂ are added according to the proportion to obtain the similarity result β:

If it is greater than the preset similarity result threshold, the candidate key point is the calibrated key point of the face image in the next frame; if it is less than the preset similarity result threshold, search in the preset key point search area until it obtains a key point greater than The key points of the threshold are used as the calibrated key points of the next frame of face image.

Preferably, it also includes: if there is still no point higher than the preset threshold after searching the complete search area, then all points in the search area are sorted by the similarity with the candidate key points, and the similarity is ranked first. as a key point after calibration.

Preferably, the DAN neural network model utilizes CNN to complete supervised learning according to the marked face key point coordinate data.

Preferably, the calculation of the gradient size and gradient direction of the gradient feature vector generation region includes the following steps:

The gradient size is divided into the horizontal gradient size G _x and the longitudinal gradient size G _y as:

G _x (x,y)=I ₂ (x+1,y)-I ₂ (x,y)

G _y (x,y)=I ₂ (x,y+1)-I ₂ (x,y)

In the formula, (x, y) is the pixel coordinate; I ₂ is the brightness of the image after Gamma correction;

The magnitude of the gradient is:

The direction of the gradient is:

Preferably, the acquisition of the regional gradient feature vector includes the following steps:

Assign the same weight to the pixels with the same distance from the center point of the gradient feature vector generation region, multiply the gradient vector of the pixel with the corresponding weight to obtain the weighted pixel gradient vector, and add the weighted pixel gradient vector to obtain the regional gradient feature vector.

Preferably, the similarity calculation formula adopts a cosine similarity calculation formula.

Preferably, the similarity result α ₁ and the similarity result α ₂ are added in proportion to obtain the similarity result β, and the formula is as follows:

β=w ₁ α ₁ +w ₂ α ₂ , w ₁ +w ₂ =1

Among them, the threshold value of β is set to 0.95.

Preferably, the searching in the preset key point search area includes the following steps:

With the candidate key point as the center, establish a 15*15 key point search area; start the search with the candidate key point as the origin, first search horizontally, then vertically, until there is a point higher than the threshold, that is, the next frame of people keypoints of the face image.

Beneficial effects of the present invention:

The invention proposes a calibration method for calibrating facial key points of a face image, which improves the anti-noise effect of the key points of the face, eliminates the jitter of the key points, improves the positioning accuracy of the key points, and can complete the task of micro-expression classification .

Description of drawings

Fig. 1 is the method flow chart of the embodiment of the present invention;

2 is a schematic diagram of the location of key points of a human face according to an embodiment of the present invention;

3 is a flow chart of HOG feature extraction according to an embodiment of the present invention

Fig. 4 (a) is the subject's face map without Gamma processing according to the embodiment of the present invention;

Fig. 4(b) is the processed face map of the tested subject according to the embodiment of the present invention;

5 is a schematic diagram of a gradient vector generation and comparison process in the vicinity of a key point according to an embodiment of the present invention;

6 is a local convolution operation and a result comparison diagram of an embodiment of the present invention;

Fig. 7 (a) is the continuous frame face image of the embodiment of the present invention;

Fig. 7 (b) is a schematic diagram of uncalibrated face key point positioning in the continuous face image of the embodiment of the present invention;

Fig. 7(c) is a schematic diagram of the location of calibrated key points of faces in continuous face images according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Example 1

A calibration method for calibrating facial key points in a face image of the present invention is shown in FIG. 1 ,

S1: Use the DAN neural network to locate the key points of the first image of the face sequence:

The dataset used is 300W, and the 300W dataset is a relatively common face alignment dataset with about 4000 subjects' images. Each subject contains multiple faces, but only one face image per batch is annotated with 68 keypoints.

DAN is a kind of similar cascaded neural network, but different from other cascaded models, its input is the whole face image, DAN can effectively overcome the problems caused by head posture and achieve better detection results. DAN needs to use CNN to complete supervised learning according to the marked face key point coordinate data. DAN is a multi-layer network. The input of the first layer is the face key point template and the original image, and the input is sent to the feedforward neural network. The feedforward neural network consists of ten convolutional layers, five pooling layers, one Fully connected layer. Each two convolutional layers is followed by a pooling layer, the activation function uses the ReLU function, and the output is the deviation value of each position of the key points of the face. The second layer is the pose estimation network, which mainly calculates the pose direction of the face in the image, and at the same time, performs affine transformation according to the standard face key point model, and proposes and straightens the face in the image. The third layer is keypoint heatmap generation. The key point heat map is a probability map of the area near each key point in the image (the selected area is 25*25). The pixel value of the image ranges from 0 to 1. The farther the position is from the key point, the smaller the pixel value. The location-based distribution of heatmaps is modeled with a Gaussian distribution. After training, the resulting model can predict the keypoints of face images. as shown in picture 2.

S2: Based on the coordinates of the key points of the first face image obtained through the DAN network, the key points of the face images of consecutive frames are calibrated sequentially, including the following steps:

S2.1: Grayscale face images of consecutive frames and perform Gamma correction.

Image grayscale converts color images into grayscale images. The grayscaled data occupies less memory and operates faster. At the same time, after converting to grayscale images, it can enhance the visual contrast effect and highlight the key to the face. area near the point. The formula for grayscale is as follows:

I ₀ =0.3I _r +0.59I _g +0.11I _b

The coefficients in this formula come from how sensitive the human eye is to red, green, and blue. It can be quickly implemented using the related functions in opencv. The purpose of Gamma correction is to adjust the contrast of the image, reduce the influence of local shadows and illumination changes in the image, and also suppress the interference of noise. To perform the Gamma operation, the image pixels need to be normalized first. The normalization formula is as follows:

I ₁ =I ₀ /255

Gamma compensation is performed on the normalized pixels. The formula of Gamma compensation is as follows:

I ₂ =I ₁ ^gamma

Perform the inverse normalization operation to obtain the processed image, as shown in Figure 4. Figure 4(a) is the face image of the subject without Gamma processing, and Figure 4(b) is the processed subject. face map.

S2.2: Perform HOG feature extraction. The process is shown in Figure 3. The calibrated key point coordinates in the previous frame of the face image are used as the coordinates of the candidate key points in the next frame, and the gradients of the two frames of images are established respectively. Feature vector generation area.

S2.3: Obtain the regional gradient feature vector by calculating the gradient size and gradient direction of the gradient feature vector generation region, and calculate the similarity result α ₁ of the regional gradient feature vectors of the two frames of face images before and after. Specifically, it includes:

The gradient size is divided into the horizontal gradient size G _x and the longitudinal gradient size G _y as:

G _x (x,y)=I ₂ (x+1,y)-I ₂ (x,y)

G _y (x,y)=I ₂ (x,y+1)-I ₂ (x,y)

In the formula, (x, y) is the pixel coordinates;

The magnitude of the gradient is:

The direction of the gradient is:

With each key point as the center, a gradient feature vector generation area with a size of 5*5 is established. Each key point will generate a regional gradient feature vector according to its corresponding gradient feature vector: the gradient feature vector is generated for the region with the same distance from the center point. The pixels are assigned the same weight, the gradient vector of the pixel is multiplied by the corresponding weight to obtain the weighted pixel gradient vector, and the weighted pixel gradient vector is added to obtain the regional gradient feature vector.

The similarity calculation formula adopts the cosine similarity calculation formula:

The gradient feature vector contrast value α ₁ is obtained. The schematic diagram of the gradient vector generation and comparison process in the vicinity of the key point is shown in Figure 5.

S2.4: Establish the convolution feature comparison area of the two frames of images with the key points as the center, perform multiple convolutions on the convolution feature comparison area to obtain the regional convolution features, and calculate the regional convolution in the face images of the front and rear frames. Feature similarity result α ₂ : With each key point as the center, a convolutional feature comparison area with a size of 7*7 is established. Three convolutions are performed in the convolution contrast area, and the size of the convolution kernel is 3*3. The local convolution operation and the result comparison are shown in Figure 6.

S2.5: Add the similarity result α ₁ and the similarity result α ₂ according to the proportion to obtain the similarity result β: The specific formula is as follows:

β=w ₁ α ₁ +w ₂ α ₂ , w ₁ +w ₂ =1

Set the β threshold to 0.95. The following is a detailed description of the key point comparison process:

S2.6: Key point comparison process:

Note the coordinates (x ₁ , y ₁ ) of _a key point in the _first face picture, and calculate the gradient feature vector similarity, convolution feature similarity and The final similarity result β ₀ . If β ₀ is greater than 0.95, it can be considered that the features near (x ₁ , y ₁ ) of the second face image are sufficiently similar to the features near (x ₁ , y ₁ ) of the first face image, that is, (x ₁ , y ₁ ) of the first face image is the same point as (x ₁ , y ₁ ) of the second face image, and (x ₁ , y ₁ ) is also the key to the second face image point. If the comparison result is lower than the threshold, a 15*15 key point search area is established with (x ₁ , y ₁ ) as the center. Start the search with (x ₁ , y ₁ ) as the origin, first search horizontally, then vertically, until a point (x ₂ , y ₂ ) higher than the threshold appears,

(x ₂ , y ₂ ) is the key point of the second face image. If there is no point higher than the threshold after searching the entire search area, then all points in the search area are sorted by their similarity to (x ₁ , y ₁ ), and the first point of similarity (x _n , y _n ) as key points. Repeat the above process for all keypoints to get the corrected keypoints. The specific effect is shown in Figure 7. Figure 7(a) is a continuous face image, Figure 7(b) is an uncalibrated face key point in the continuous face image, and Figure 7(c) is a calibrated face in the continuous image. face key points.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention. Inside.

Claims (8)

1. a calibration method of face image facial key point calibration, is characterized in that, comprises: Based on the coordinates of the key points of the first face image obtained through the trained DAN network, the key points of the face image in the next frame are calibrated in turn, including the following steps: Grayscale the face images of consecutive frames and perform Gamma correction; The calibrated key point coordinates in the face image of the previous frame are used as the coordinates of the candidate key points of the next frame, and the gradient feature vector generation regions of the two frames of images are respectively established around them; By calculating the gradient magnitude and gradient direction of the gradient feature vector generation region, the regional gradient feature vector is obtained, and the similarity result α ₁ of the regional gradient feature vectors of the two frames of face images before and after is calculated; The convolution feature comparison area of the two frames of images before and after is established with the candidate key points as the center, and the convolution feature comparison area is convolved multiple times to obtain the regional convolution feature, and the regional convolution in the two frames of face images before and after is calculated. Feature similarity result α ₂ ; The similarity result α ₁ and the similarity result α ₂ are added according to the proportion to obtain the similarity result β: If it is greater than the preset similarity result threshold, the candidate key point is the calibrated key point of the face image in the next frame; if it is less than the preset similarity result threshold, search in the preset key point search area until it obtains a key point greater than The key points of the threshold are used as the calibrated key points of the next frame of face image. 2. the calibration method of face image facial key point demarcation according to claim 1, is characterized in that, also comprises: if still does not have the point higher than preset threshold value after searching the complete piece search area, then will search area. All points are sorted by similarity with candidate key points, and the point with the first similarity is used as the calibrated key point. 3. The calibration method of face image facial key point demarcation according to claim 1, is characterized in that, described DAN neural network utilizes CNN to complete supervision learning afterward according to the marked face key point coordinate data. 4. the calibration method of face image face key point calibration according to claim 1, is characterized in that, the calculation of the gradient size and gradient direction of described gradient feature vector generation area, comprises the following steps: The gradient size is divided into the horizontal gradient size G _x and the longitudinal gradient size G _y as: G _x (x,y)=I ₂ (x+1,y)-I ₂ (x,y) G _y (x,y)=I ₂ (x,y+1)-I ₂ (x,y) In the formula, (x, y) is the pixel coordinate; I ₂ is the brightness of the image after Gamma correction; The magnitude of the gradient is:

The direction of the gradient is:

5. the calibration method of human face image face key point calibration according to claim 4, is characterized in that, the acquisition of described regional gradient feature vector, comprises the following steps: Assign the same weight to the pixels with the same distance from the center point of the gradient feature vector generation region, multiply the gradient vector of the pixel with the corresponding weight to obtain the weighted pixel gradient vector, and add the weighted pixel gradient vector to obtain the regional gradient feature vector. 6 . The calibration method for face key point calibration in a human face image according to claim 1 , wherein the similarity calculation formula adopts a cosine similarity calculation formula. 7 . 7. the calibration method of face image face key point demarcation according to claim 1, is characterized in that, described similarity result α ₁ and similarity result α ₂ are added by proportion, obtain similarity result β, formula as follows: β=w ₁ α ₁ +w ₂ α ₂ , w ₁ +w ₂ =1 Among them, the threshold value of β is set to 0.95. 8. the calibration method of face key point calibration of human face image according to claim 1, is characterized in that, described searching in the preset key point search area, comprises the following steps: With the candidate key point as the center, establish a 15*15 key point search area; start the search with the candidate key point as the origin, first search horizontally, then vertically, until there is a point higher than the threshold, that is, the next frame of people keypoints of the face image.

Images