CN108229268A - Expression recognition and convolutional neural network model training method and device and electronic equipment - Google Patents

Abstract

The embodiment of the invention provides an expression recognition and convolutional neural network model training method, a device and electronic equipment, wherein the method comprises the following steps: extracting facial expression features of the facial image to be detected through a convolution layer part of the convolution neural network model and the acquired facial key points in the facial image to be detected to obtain a facial expression feature map; determining ROI (region of interest) corresponding to each face key point in the face expression feature map; performing pooling treatment on each determined ROI through a pooling layer part of the convolutional neural network model to obtain a pooled ROI feature map; and obtaining an expression recognition result of the face image at least according to the ROI characteristic diagram. By the embodiment of the invention, fine expression changes can be effectively captured, differences caused by different facial postures can be better processed, and the detailed information of changes of a plurality of areas of the face is fully utilized, so that the fine expression changes and faces in different postures can be more accurately identified.

Description

Expression recognition and convolutional neural network model training method, device and electronic equipment

technical field

The embodiments of the present invention relate to the technical field of artificial intelligence, and in particular to an expression recognition method, device and electronic equipment, and a convolutional neural network model training method, device and electronic equipment.

Background technique

Facial expression recognition technology refers to assigning an expression category to a given face image, including: anger, disgust, happiness, sadness, fear, surprise, etc. At present, facial expression recognition technology has gradually shown broad application prospects in the fields of human-computer interaction, clinical diagnosis, distance education, investigation and interrogation, and is a popular research direction of computer vision and artificial intelligence.

An existing facial expression recognition technology is a recognition technology based on a traditional machine learning framework. Expression recognition using this traditional machine learning framework can include four basic steps: face detection, face feature extraction, feature dimensionality reduction, and feature classification. However: first, face feature extraction requires manual design and extraction, which requires expertise in a specific field; second, compared with deep feature (feature map), classic geometric features such as Gabor filter, SIFT, etc. Weak; Third, traditional machine learning methods are difficult to use more and more training data, the training time is long, and the training process is scattered and complicated.

As a result, the cost of existing expression recognition is high, and the accuracy of expression recognition is low.

Contents of the invention

An embodiment of the present invention provides a technical solution for facial expression recognition.

According to the first aspect of an embodiment of the present invention, a method for facial expression recognition is provided, including: using the convolutional layer part of the convolutional neural network model and the key points of the human face in the acquired human face image to be detected, the facial expression to be detected Face image carries out facial expression feature extraction, obtains facial expression feature map; Determines the region of interest ROI corresponding to each face key point respectively in described facial expression feature map; Through the pooling layer of convolutional neural network model Partially performing pooling processing on each of the determined ROIs to obtain a pooled ROI feature map; at least acquiring an expression recognition result of the face image according to the ROI feature map.

Optionally, the face image includes a static face image.

Optionally, the face image includes a face image in a sequence of video frames.

Optionally, obtaining the facial expression recognition result of the face image at least according to the ROI feature map includes: acquiring the face image of the current frame according to the ROI feature map of the face image of the current frame. Preliminary facial expression recognition results; according to the preliminary facial expression recognition results of the current frame and the facial expression recognition results of at least one previous frame of human facial images, the facial expression recognition results of the current frame's facial images are obtained.

Optionally, according to the preliminary expression recognition result of the current frame and the expression recognition result of the face image of at least one previous frame, acquiring the expression recognition result of the face image of the current frame includes: The preliminary facial expression recognition result of the human face image and the facial expression recognition result of at least one previous frame of the human face image are weighted to obtain the expression recognition result of the human face image of the current frame, wherein the current The weight of the preliminary expression recognition result of the face image of the frame is greater than the weight of the expression recognition result of any previous frame of the face image.

Optionally, according to the preliminary expression recognition result of the current frame and the expression recognition result of the face image of at least one previous frame, before acquiring the expression recognition result of the face image of the current frame, it also includes: determining the The position of the current frame in the video frame sequence is greater than or equal to the set position threshold.

Optionally, the method further includes: in response to the position of the current frame in the sequence of video frames being smaller than a set position threshold, outputting the facial expression recognition result of the face image of the current frame, and/or Or, save the facial expression recognition result of the facial image of the current frame.

Optionally, through the convolution layer part of the convolutional neural network model and the key points of the face in the face image to be detected, the facial expression feature extraction is performed on the face image to be detected to obtain the facial expression A feature map, comprising: performing face key point detection on a face image to be detected to obtain the face key points in the face image; according to the face key points, through the convolution of the convolutional neural network model The layer part performs facial expression feature extraction on the face image to obtain a facial expression feature map.

Optionally, before performing facial expression feature extraction on the face image to be detected through the convolutional layer part of the convolutional neural network model and the acquired face key points in the face image to be detected, the The method also includes: acquiring a sample image for training, and using the sample image to train the convolutional neural network model, wherein the sample image includes information about key points of a human face and labeling information of facial expressions.

Optionally, the acquiring a sample image for training, and using the sample image to train the convolutional neural network model includes: acquiring a sample image for training, and using the convolutional layer part of the convolutional neural network model to The sample image is subjected to facial expression feature extraction to obtain a facial expression feature map; determine the region of interest ROI corresponding to each face key point in the facial expression feature map; through the pooling layer part of the convolutional neural network model performing pooling processing on each determined ROI to obtain a pooled ROI feature map; adjusting network parameters of the convolutional neural network model at least according to the ROI feature map.

Optionally, determining the region of interest (ROI) corresponding to each key point in the facial expression feature map includes: determining the corresponding ROI according to the coordinates of each key point in the facial expression feature map. Each position of each of the determined positions is used as a reference point to acquire corresponding areas of each set range, and each acquired area is determined as a corresponding ROI.

Optionally, performing pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model, and obtaining the ROI feature map after pooling includes: determining Each ROI of each ROI is pooled to obtain a pooled ROI feature map of a set size; said adjusting the network parameters of the convolutional neural network model at least according to the ROI feature map includes: setting the The size of the ROI is input into the loss layer to obtain the error of the expression classification result of the expression classification of the sample image; according to the error of the expression classification result, the network parameters of the convolutional neural network model are adjusted.

Optionally, inputting the ROI of the set size into the loss layer, and obtaining the expression classification result error of the expression classification of the sample image includes: inputting the ROI of the set size into the loss layer, passing through the loss layer The logistic regression loss function calculates the expression classification result error and outputs it.

Optionally, the logistic regression loss function is a logistic regression loss function with a set number of expression classifications.

Optionally, the sample images of human facial expressions to be trained are sample images of video frame sequences.

Optionally, before the acquisition of the sample image for training and the corresponding key point information of the human face, the method further includes: detecting the sample image for training to obtain the key point information of the human face.

According to a second aspect of an embodiment of the present invention, there is provided an expression recognition device, including: a first determination module, which is used to pass the convolutional layer part of the convolutional neural network model and the acquired human face image to be detected Face key point, the face image to be detected carries out facial expression feature extraction, obtains facial expression feature map; The second fifth determining module is used to determine that in the described human face expression feature map, it corresponds to each key point of human face respectively ROI of the region of interest; the third determination module is used to pool the determined ROIs through the pooling layer part of the convolutional neural network model to obtain the ROI feature map after pooling; the fourth determination module is used to Acquiring an expression recognition result of the face image at least according to the ROI feature map.

Optionally, the face image includes a static face image.

Optionally, the face image includes a face image in a sequence of video frames.

Optionally, the third determination module includes: a first acquisition submodule, configured to acquire the preliminary expression recognition of the face image of the current frame according to the ROI feature map of the face image of the current frame Result; the second acquisition sub-module is used to acquire the expression recognition result of the face image of the current frame according to the preliminary expression recognition result of the current frame and the expression recognition result of the face image of at least one previous frame.

Optionally, the second acquisition submodule is configured to perform weighting processing on the preliminary facial expression recognition result of the face image of the current frame and the facial expression recognition result of at least one previous frame of the face image, The expression recognition result of the face image of the current frame is obtained, wherein the weight of the preliminary expression recognition result of the face image of the current frame is greater than the weight of the expression recognition result of any previous frame of the face image.

Optionally, the device further includes: a fifth determining module, configured to determine that the position of the current frame in the video frame sequence is greater than or equal to a set position threshold.

Optionally, the device further includes: a response module, configured to output the facial expression of the facial image of the current frame in response to the position of the current frame in the sequence of video frames being smaller than a set position threshold The recognition result, and/or, saving the facial expression recognition result of the face image of the current frame.

Optionally, the first determination module is configured to perform face key point detection on the face image to be detected, and obtain face key points in the face image; according to the face key points, through the The convolutional layer part of the convolutional neural network model performs facial expression feature extraction on the face image to obtain a facial expression feature map.

Optionally, the first determination module is configured to extract face key points from the face image to be detected through the convolutional layer part of the convolutional neural network model; according to the extracted face key points, the face image to be detected The facial expression feature extraction is performed on the face image, and the facial expression feature map is obtained.

Optionally, the device further includes: a training module, configured to acquire a sample image for training, and use the sample image to train the convolutional neural network model, wherein the sample image contains information about key points of a human face and annotation information of facial expressions.

Optionally, the training module includes: a first sub-module, configured to obtain a sample image for training, and perform facial expression feature extraction on the sample image through the convolutional layer part of the convolutional neural network model to obtain human Facial expression feature map; The second submodule is used to determine the region of interest ROI corresponding to each face key point in the described human facial expression feature map; the third submodule is used to pass through the pool of the convolutional neural network model The pooling layer part performs pooling processing on each determined ROI to obtain a pooled ROI feature map; the fourth submodule is used to adjust the network parameters of the convolutional neural network model at least according to the ROI feature map.

Optionally, the second submodule is configured to determine corresponding positions in the facial expression feature map according to the coordinates of each key point of the face; using the determined positions as reference points, obtain the corresponding For each area within the set range, each acquired area is determined as a corresponding ROI.

Optionally, the third submodule is configured to perform pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model, and obtain a pooled ROI feature map of a set size; the first The four sub-modules are used to input the ROI of the set size into the loss layer, and obtain the error of the expression classification result of the expression classification of the sample image; according to the error of the expression classification result, adjust the convolutional neural network model. Network parameters.

Optionally, the fourth sub-module is configured to input the ROI of the set size into a loss layer, and calculate and output the expression classification result error through a logistic regression loss function of the loss layer.

Optionally, the logistic regression loss function is a logistic regression loss function with a set number of expression classifications.

Optionally, the sample images of human facial expressions to be trained are sample images of video frame sequences.

Optionally, the device further includes: a sixth determination module, configured to detect the sample images used for training to obtain information about key points of human faces.

According to a third aspect of the embodiments of the present invention, there is provided an electronic device, including: a processor, a memory, a communication element, and a communication bus, and the processor, the memory, and the communication element complete mutual communication via the communication bus. communication among them; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute any one of the facial expression recognition methods in the first aspect.

According to a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, and the computer-readable storage medium stores: the convolutional layer part and the obtained human to be detected through the convolutional neural network model The key points of the face in the face image, the face image to be detected is extracted to obtain the executable instruction of the facial expression feature map; it is used to determine the relationship between the facial expression feature map and each face key point Executable instructions for corresponding regions of interest ROIs; for performing pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model, and obtaining executable instructions for pooled ROI feature maps; for An executable instruction for obtaining an expression recognition result of the face image at least according to the ROI feature map.

According to a fifth aspect of an embodiment of the present invention, a method for training a convolutional neural network model is provided, including: acquiring a sample image for training and information about key points of a human face, wherein the sample image includes a human face The labeling information of expression; Carry out facial expression feature extraction to described sample image by the convolutional layer part of convolutional neural network model, obtain facial expression characteristic map; Determine described human facial expression characteristic map and each face key point Respectively corresponding to the ROI of the region of interest; through the pooling layer part of the convolutional neural network model, pooling is performed on each determined ROI to obtain a pooled ROI feature map; at least according to the ROI feature map, adjust the volume The network parameters of the product neural network model.

Optionally, determining the region of interest (ROI) corresponding to each key point in the facial expression feature map includes: determining the corresponding ROI according to the coordinates of each key point in the facial expression feature map. Each position of each of the determined positions is used as a reference point to acquire corresponding areas of each set range, and each acquired area is determined as a corresponding ROI.

Optionally, performing pooling processing on each of the determined ROIs through the pooling layer part of the convolutional neural network model to obtain a pooled ROI feature map includes: Each of the determined ROIs is pooled to obtain a pooled ROI feature map of a set size; the adjustment of the network parameters of the convolutional neural network model at least according to the ROI feature map includes: setting the The fixed-sized ROI is input into the loss layer to obtain the error of the expression classification result of the expression classification of the sample image; according to the error of the expression classification result, the network parameters of the convolutional neural network model are adjusted.

Optionally, inputting the ROI of the set size into the loss layer, and obtaining the expression classification result error of the expression classification of the sample image includes: inputting the ROI of the set size into the loss layer, passing through the loss layer The logistic regression loss function calculates the expression classification result error and outputs it.

Optionally, the logistic regression loss function is a logistic regression loss function with a set number of expression classifications.

Optionally, the sample images of human facial expressions to be trained are sample images of video frame sequences.

Optionally, before the acquisition of the sample image for training and the corresponding key point information of the human face, the method further includes: detecting the sample image for training to obtain the key point information of the human face.

According to the sixth aspect of the embodiments of the present invention, a convolutional neural network model training device is provided, including: a first acquisition module, configured to acquire training sample images and corresponding face key point information, wherein the The sample image contains labeling information of human facial expression; the second acquisition module is used to extract the facial expression feature of the sample image through the convolutional layer part of the convolutional neural network model, and obtain the facial expression feature map; the second Three acquisition modules, for determining the region of interest ROI corresponding to each face key point in the facial expression feature map; the fourth acquisition module, for determining by the pooling layer part of the convolutional neural network model Each ROI is pooled to obtain a pooled ROI feature map; a fifth acquisition module is configured to adjust network parameters of the convolutional neural network model at least according to the ROI feature map.

Optionally, the third acquisition module is configured to determine corresponding positions in the facial expression feature map according to the coordinates of each key point of the face; using the determined positions as reference points, acquire the corresponding For each area within the set range, each acquired area is determined as a corresponding ROI.

Optionally, the fourth acquisition module is configured to perform pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model, and obtain a pooled ROI feature map of a set size; the first Five acquisition modules, including: a first acquisition sub-module, used to input the ROI of the set size into the loss layer, and obtain the expression classification result error of the expression classification of the sample image; Expression classification result error, adjust the network parameters of the convolutional neural network model.

Optionally, the first acquisition sub-module is configured to input the ROI of the set size into a loss layer, and calculate and output the expression classification result error through a logistic regression loss function of the loss layer.

Optionally, the logistic regression loss function is a logistic regression loss function with a set number of expression classifications.

Optionally, the sample images of human facial expressions to be trained are sample images of video frame sequences.

Optionally, the device further includes: a sixth acquisition module, configured to detect the sample images used for training to obtain information about key points of human faces.

According to a seventh aspect of the embodiments of the present invention, there is provided an electronic device, including: a processor, a memory, a communication element, and a communication bus, and the processor, the memory, and the communication element complete mutual communication through the communication bus communication among them; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute any one of the convolutional neural network model training methods in the fifth aspect.

According to an eighth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, and the computer-readable storage medium stores: information for obtaining training sample images and corresponding face key points, wherein, The sample image contains executable instructions for labeling information of human facial expressions; it is used to extract the facial expression features of the sample images through the convolutional layer part of the convolutional neural network model to obtain the human facial expression feature map. Executing instructions; executable instructions for determining the region of interest ROI corresponding to each face key point in the facial expression feature map; for each ROI determined by the pooling layer part of the convolutional neural network model An executable instruction for performing pooling processing to obtain a pooled ROI feature map; an executable instruction for adjusting network parameters of the convolutional neural network model at least according to the ROI feature map.

According to the technical solution provided by the embodiment of the present invention, after the facial expression feature extraction is performed according to the key points of the human face, and the facial expression feature map is obtained, the corresponding key points of the human face are determined from the facial expression feature map according to the key points of the human face. ROI (Region Of Interest, region of interest), after the ROI is processed by the ROI pooling layer, the ROI feature map can be obtained; then, the facial expression is determined according to the ROI feature map. By selecting the area corresponding to the key points of the face as the ROI, it is possible to effectively capture subtle expression changes, and at the same time better handle the differences caused by different facial poses, and make full use of the detailed information of the changes in multiple areas of the face. Subtle expression changes and faces with different poses are more accurately recognized.

Description of drawings

FIG. 1 is a flow chart of the steps of an expression recognition method according to Embodiment 1 of the present invention;

FIG. 2 is a flow chart of the steps of an expression recognition method according to Embodiment 2 of the present invention;

FIG. 3 is a flowchart of steps of an expression recognition method according to Embodiment 3 of the present invention;

4 is a flow chart of steps of a method for training a convolutional neural network model according to Embodiment 4 of the present invention;

5 is a structural block diagram of an expression recognition device according to Embodiment 5 of the present invention;

6 is a structural block diagram of a convolutional neural network model training device according to Embodiment 6 of the present invention;

FIG. 7 is a schematic structural diagram of an electronic device according to Embodiment 7 of the present invention;

FIG. 8 is a schematic structural diagram of an electronic device according to Embodiment 8 of the present invention.

Detailed ways

The specific implementation manners of the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings (the same symbols in several drawings indicate the same elements) and the embodiments. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

Those skilled in the art can understand that terms such as "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc. necessary logical sequence.

Embodiment one

Referring to FIG. 1 , it shows a flowchart of steps of an expression recognition method according to Embodiment 1 of the present invention.

The facial expression recognition method of the present embodiment comprises the following steps:

Step S102: Using the convolutional layer part of the convolutional neural network model and the acquired face key points in the face image to be detected, perform facial expression feature extraction on the face image to be detected to obtain a face expression feature map.

The trained convolutional neural network model has the function of facial expression recognition, which at least includes an input layer part, a convolutional layer part, a pooling layer part, a fully connected layer part, etc. Among them, the input layer part is used to input the image; the convolution layer part performs feature extraction; the pooling layer part performs pooling processing on the processing results of the convolution layer part, such as downsampling the feature map obtained by the convolution layer part, etc. ; The fully connected layer part can be used for classification, etc.

In this embodiment, facial expression feature extraction is performed through the convolutional layer part of the convolutional neural network model to obtain a facial expression feature map. In addition, for the acquisition of face key points, in a feasible way, before inputting the convolutional neural network model, the face key point detection can be performed on the face image to be detected; in another feasible way, It can be extracted by the convolutional layer part of the convolutional neural network model, that is, the convolutional layer part first extracts the face key points in the face image to be detected, and then performs further human face key points based on the extracted face key points. Facial expression feature extraction to obtain the facial expression feature map; in another feasible way, before inputting the convolutional neural network model, the human face image to be detected can be manually annotated to obtain the key points of the face.

Step S104: Determine the ROIs in the facial expression feature map corresponding to each key point of the face.

The facial expression feature map output by the convolutional layer contains the processing results of the entire image. The processing results contain a large amount of data. If facial expression recognition is based on this, a large amount of data needs to be processed, and the system processing burden is heavy. .

To this end, in the solution of the embodiment of the present invention, firstly, according to the key points of the face, the ROI (Region Of Interest) corresponding to each key point of the face is determined. For example, when determining the ROI corresponding to each face key point according to the information of the key points of the face, in the facial expression feature map, determine the corresponding position according to the coordinates of the key points of the face; the determined position is In the center, acquire the area of the set range, and determine the acquired area as the ROI.

Step S106: Perform pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model, and obtain a pooled ROI feature map.

Wherein, pooling processing includes but not limited to down-sampling processing.

Step S108: Obtain an expression recognition result of the face image at least according to the ROI feature map.

The ROI feature map contains feature information of human facial expressions, and an expression recognition result of human facial expressions in the human face image to be detected can be obtained according to the ROI feature map.

Through this embodiment, after performing facial expression feature extraction according to the key points of the human face and obtaining the facial expression feature map, the ROI corresponding to the key points of the human face is determined from the facial expression feature map according to the key points of the human face. After the processing of the ROI pooling layer, the ROI feature map can be obtained; then, the facial expression is determined according to the ROI feature map. By selecting the area corresponding to the key points of the face as the ROI, it is possible to effectively capture subtle expression changes, and at the same time better handle the differences caused by different facial poses, and make full use of the detailed information of the changes in multiple areas of the face. Subtle expression changes and faces with different poses are more accurately recognized.

Embodiment two

Referring to FIG. 2 , it shows a flowchart of steps of an expression recognition method according to Embodiment 2 of the present invention.

In this embodiment, a convolutional neural network model with a facial expression recognition function is trained first, and then facial expression recognition of images is performed based on the model. However, those skilled in the art should understand that in actual use, the convolutional neural network model trained by a third party can also be used for facial expression recognition.

The facial expression recognition method of the present embodiment comprises the following steps:

Step S202: Obtain sample images for training, and use the sample images to train a convolutional neural network model.

Wherein, the sample image may be a static image, or a sample image of a sequence of video frames. The sample image contains the information of the key points of the human face and the annotation information of the facial expression. In this embodiment, the information of the key points of the human face is obtained by detecting the sample images used for training.

In a feasible way of realizing this step, the sample image used for training is obtained, and the facial expression feature extraction is performed on the sample image through the convolution layer part of the convolutional neural network model to obtain the facial expression feature map; determine the facial expression The ROIs corresponding to the key points of each face in the feature map; the pooling layer part of the convolutional neural network model is used to pool the determined ROIs to obtain the pooled ROI feature map; at least according to the ROI feature map, Adjust the network parameters of the convolutional neural network model.

Wherein, in a feasible manner, determining the ROI corresponding to each key point in the facial expression feature map includes: determining the corresponding positions according to the coordinates of each key point in the facial expression feature map ; Using each determined position as a reference point, acquire corresponding areas of each set range, and determine each acquired area as a corresponding ROI.

Through the pooling layer part of the convolutional neural network model, pooling is performed on each determined ROI, and a pooled ROI feature map of a set size can be obtained. When adjusting the network parameters of the convolutional neural network model according to the ROI feature map, the ROI feature map of the set size can be input into the loss layer to obtain the error of the expression classification result of the expression classification of the sample image; according to the error of the expression classification result, Adjust the network parameters of the convolutional neural network model. Wherein, the adjusted network parameters include but not limited to weight parameter weight, bias parameter bias and so on.

The error of the expression classification result can be obtained by inputting the ROI of the set size into the loss layer, and calculating and outputting the error of the expression classification result through the logistic regression loss function of the loss layer. Wherein, the logistic regression loss function may be a logistic regression loss function with a set number of expression classifications.

Through the above process, the training of the convolutional neural network model with expression recognition function is realized, and then, the convolutional neural network model based on the training can be used for facial expression detection.

Step S204: Obtain the face image to be detected.

Wherein, the face image to be detected may be a static face image, or a face image of a sequence of video frames.

Step S206: Perform face key point detection on the face image to be detected, and obtain face key points in the face image.

In this embodiment, a method of first performing face image detection to obtain face key points is adopted. But as mentioned above, if the convolutional neural network model has the function of face key point detection, the face image to be detected can be directly input into the convolutional neural network model, and the convolutional layer part of the convolutional neural network model is treated The detected face image is subjected to face key point extraction. Furthermore, according to the extracted face key points, the facial expression feature extraction is performed on the face image to be detected, and the facial expression feature map is obtained.

Step S208: Using the convolutional layer part of the convolutional neural network model and the acquired face key points in the face image to be detected, perform facial expression feature extraction on the face image to be detected to obtain a face expression feature map.

Step S210: Determine ROIs in the facial expression feature map corresponding to each key point of the face.

Step S212: Perform pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model, and obtain a pooled ROI feature map corresponding to each ROI.

Step S214: Obtain an expression recognition result of the face image at least according to the ROI feature map.

After obtaining the ROI feature map, expression recognition can be performed according to the ROI feature map.

In a preferred solution, when using the convolutional neural network model to detect facial expression images in the continuous sequence of video frames, if the current frame is used as a benchmark, the convolutional neural network model can be firstly used to detect the facial expression images in the sequence of video frames. The current frame is detected, and according to the ROI feature map of the face image of the current frame, the preliminary expression recognition result of the face image of the current frame is obtained; and then, according to the preliminary expression recognition result of the current frame and the face image of at least one previous frame The expression recognition result of the face image of the current frame is obtained. For example, after obtaining the preliminary facial expression recognition result of the current frame, it can also be judged whether the position of the current frame in the video frame sequence is greater than or equal to the set position threshold; If the position is less than the set position threshold, the preliminary facial expression recognition result of the current frame is output as the final facial expression recognition result of the facial image of the current frame, and/or, the facial expression of the facial image of the current frame is saved Recognition result; If so, then obtain the facial expression recognition result of the video frame of setting quantity before the current frame; Combine the preliminary expression recognition result of the human face image of the current frame with the person of the human face image of at least one previous frame obtained The facial expression recognition result is linearly weighted to obtain the facial expression recognition result of the face image of the current frame. Wherein, at least one previous frame may be one or more consecutive frames before the current frame, or may be one or more discontinuous frames before the current frame. Through the above process, the expression recognition result of the current frame can be determined according to the detection results of consecutive multiple frames, avoiding the error of single frame detection, and making the detection result more accurate.

Wherein, when performing linear weighting processing on the facial expression recognition result of the current frame and the facial expression recognition result of at least one previous frame obtained, it can be the preliminary facial expression recognition result of the current frame and the obtained previous frame The facial expression recognition results are respectively set with weights. When setting the weights, the weight of the preliminary facial expression recognition results of the current frame is greater than the weight of the facial expression recognition results of any previous frame obtained; then, according to the set weights, the The preliminary facial expression recognition result of the current video frame is linearly weighted with the obtained facial expression recognition result of the previous frame. Because the expression recognition is mainly performed on the current video frame, a heavier weight is set for the detection result of the current video frame. While the detection result of the associated video frame is used as a reference, it can effectively ensure that the current video frame is used as the detection target.

It should be noted that, in the above process, the set position threshold, the set number of video frames before the current frame, and the set weights can all be appropriately set by those skilled in the art according to the actual situation. Wherein, preferably, the set number of video frames preceding the current video frame is three.

Through this embodiment, the use of a convolutional neural network model capable of accurately recognizing human facial expressions can capture subtle changes in facial expressions, making expression recognition more accurate and faster. Moreover, for a continuous video frame sequence, by fusing the detection results of multiple consecutive frames, the error of single frame detection is effectively avoided, and the accuracy of expression detection is further improved.

Embodiment three

Referring to FIG. 3 , it shows a flowchart of steps of an expression recognition method according to Embodiment 3 of the present invention.

This embodiment describes the facial expression recognition method of the embodiment of the present invention in the form of a specific example. The facial expression recognition method of this embodiment includes both a convolutional neural network model training part and a facial expression recognition part using the trained convolutional neural network model.

The facial expression recognition method of the present embodiment comprises the following steps:

Step S302: Collect facial expression images and perform expression labeling to form a sample image set to be trained.

For example, ten expressions were manually marked: angry, calm, confused, disgusted, happy, sad, scared, surprised, squinting, and screaming.

Step S304: Use the face detection algorithm to detect the faces and their key points in each sample image, and use the key points to align the faces.

In this step, the face and its key points in each sample image can be detected using a conventional face detection algorithm, such as 21 face key points including eyes, mouth, etc.; then, the 21 face key points are used to align the faces .

Step S306: training a CNN model using the sample images labeled with expressions and the key points of the human face.

In this embodiment, a brief structure example of a CNN model is as follows:

//first part

1. Data input layer

//the second part

2.<=1 convolutional layer 1_1(3x3x4/2)

3. <= 2 nonlinear response ReLU layer

4.<=3Pooling layer//Ordinary Pooling layer

5.<=4 convolutional layer 1_2(3x3x6/2)

6. <=5 nonlinear response ReLU layer

7. <=6Pooling layer

8.<=7 convolutional layer 1_3(3x3x6)

9. <=8 nonlinear response ReLU layer

10.<=9Pooling layer

11.<=10 convolutional layer 2_1(3x3x12/2)

12. <=11 nonlinear response ReLU layer

13. <= 12Pooling layer

14.<=13 convolutional layer 2_2 (3x3x12)

15.<=14 nonlinear response ReLU layer

16.<=15Pooling layer

17.<=16 nonlinear response ReLU layer

18.<=17 convolutional layer 5_4 (3x3x16)

//the third part

19.<=18ROI Pooling layer//Pooling layer for ROI pooling

20.<=19 fully connected layers

21. <= 20 loss layers

In the above CNN model structure, the sample images and face key points that have been marked with expressions are input into the CNN model through the input layer of the first part for training; then they are processed through the conventional convolution layer part of the second part; based on the second part The processing result obtains the ROI feature map according to the key points of the face, and inputs the obtained ROI feature map into the ROI Pooling layer for ROI pooling processing to obtain the pooled ROI feature map; the pooled ROI feature map is then sequentially input into the fully connected layer and the loss layer; determine how to adjust the network parameters of the CNN model according to the processing results of the loss layer, and train the CNN model.

Among them, when the ROI feature map is obtained based on the key points of the face based on the processing results of the second part, for the ROI corresponding to the 21 key points of the face, it can be mapped back to the last volume of the CNN model according to the coordinates of the 21 key points On the feature map output by the multiplication layer (the 32nd layer in this embodiment), that is, the 21 key points of the human face detected on the original sample image are mapped to the feature map output by the 32nd layer, and these key points 21 small areas (such as 3×3 areas, or irregular areas, etc.) are cut out on the feature map centered on the point, and then the feature maps of these 21 areas are used as the input of the ROI Pooling layer to obtain the ROI feature map, and then Input the ROI feature map to the fully connected layer, followed by a 10-class logistic regression loss function layer (for example, SoftmaxWithloss Layer), calculate the result and the marked facial expression label to get the error, and backpropagate the error to update The parameters of the CNN model (including the parameters of the fully connected layer). This cycle is repeated until the error no longer decreases, the CNN model converges, and the trained model is obtained.

Because the 21 ROI areas cover all positions related to facial expressions, and there is no redundant information, the CNN model can focus more on learning these areas, and it is easier to capture subtle changes in facial muscles; the ROI Pooling layer is fixed later The length of the ROI feature representation allows the same network structure to be used when inputting ROI regions of different sizes; the fixed-length feature representation is then input into the fully connected layer and the loss layer in turn to obtain the final expression classification result.

Among them, the ROI Pooling layer is a pooling layer for the ROI feature map. For example, if the coordinates of a certain ROI area are (x1, y1, x2, y2), then the input size is (y2-y1)×(x2-x1), if the ROI The output size of the Pooling layer is pooled_height×pooled_width, then the output of each grid is the area pooling result of [(y2-y1)/pooled_height]×[(x2-x1)/pooled_width].

In addition, it should be noted that:

In the above description of the convolutional network structure, 2.<=1 indicates that the current layer is the second layer, and the input is the first layer; the brackets behind the convolutional layer are the convolutional layer parameters (3x3x16) indicating that the convolution kernel size is 3x3, and the channel The number is 16. Others can be deduced in the same way, so I won't repeat them here.

In the above convolutional network structure, each convolutional layer is followed by a nonlinear response unit ReLU. Preferably, the ReLU can use PReLU (ParametricRectified Linear Units, parametric rectified linear unit), so as to effectively improve the detection accuracy of the CNN model.

In addition, setting the convolution kernel of the convolution layer to 3x3 can better integrate local information; setting the stride interval of the convolution layer can allow the upper layer features to obtain a larger field of view without increasing the amount of calculation.

However, those skilled in the art should understand that the above-mentioned size of the convolution kernel, the number of channels, and the number of layers of the convolutional layer are all exemplary illustrations. In practical applications, those skilled in the art can adapt according to actual needs. adjustment, which is not limited in this embodiment of the present invention. In addition, the combination and parameters of all layers in the convolutional network model in this embodiment are optional and can be combined arbitrarily.

Step S308: Perform expression recognition on the aligned facial expression images through the trained CNN model, and obtain a recognition result.

The difference from CNN model training is that when using the trained CNN model for expression recognition, the fully connected layer of the CNN model is followed by a very similar logistic regression layer instead of a logistic regression loss function layer to directly obtain recognition result.

For a single image, expression recognition can be performed directly through the CNN model trained above.

For sequences of video frames, each frame is identified as a single image. However, in order to improve the accuracy of facial expression recognition in video frame sequences, multiple frames can be fused. For example, let t=1 be the start frame of the video, when t>=3, that is, when the position of the current frame is greater than or equal to the third frame, the current frame and the previous two frames are recognized simultaneously, and the recognition results of the three frames are respectively obtained. If the three input frames are recorded as Xt-2, Xt-1 and Xt respectively, and the recognition results of the three frames are recorded as Yt-2, Yt-1 and Yt, and the recognition results of these three frames are linearly weighted, set The weight of the current frame is 0.5, and the weights of the two frames before this frame are both 0.25, so the final prediction result is Y=0.25×Yt-2+0.25×Yt-1+0.5×Yt. When t<3, that is, when the current frame position is smaller than the third frame, Y=Yt. It should be clear to those skilled in the art that the weights of the above-mentioned frames are only exemplary illustrations. In practical applications, those skilled in the art can properly set the weights for each frame according to actual needs, so that the weight corresponding to the current frame is greater than that of other frames, that is, Can.

Through this embodiment, by selecting the region corresponding to the 21 key points of the human face as the ROI, making full use of the detailed information of the changes in the multiple regions of the face, it is possible to capture the subtle changes in the expression of the face, making the recognition more accurate and fast; Multiple frames are fused, so that the solution of this embodiment can be effectively applied to facial expression recognition based on video.

The facial expression recognition method in this embodiment can be executed by any suitable device with data processing capabilities, including but not limited to: mobile terminals, PCs, servers, vehicle-mounted devices, advertising machines, appearance machines, and the like.

Embodiment four

Referring to FIG. 4 , it shows a flowchart of steps of a method for training a convolutional neural network model according to Embodiment 4 of the present invention.

The convolutional neural network model training method of the present embodiment includes the following steps:

Step S402: Acquiring training sample images and information of corresponding facial key points.

Wherein, the sample image contains the annotation information of human facial expression. When performing CNN training, the sample image can be annotated in advance. In this embodiment, the corresponding facial expression information is marked in the sample image, so that the subsequent The information determines whether the CNN training results are accurate.

In addition, for each sample image, it is also necessary to obtain the information of the corresponding key points of the face. Therefore, in practical applications, before this step, it may also include: detecting the sample images used for training to obtain information about key points of the human face. In order to make the training effect better, after obtaining the information of key points of the face, face alignment can also be performed according to these key points, and the sample images after face alignment are input into CNN for sample training. Through face alignment, the training effect of sample images can be improved.

In the above process, the detection of the samples to obtain the key points of the face and the alignment of the faces can be implemented by those skilled in the art in any appropriate related manner, wherein the realization of the detection of the samples to obtain the key points of the face may include but not limited to: CNN of face key point positioning function, or ASM (ActiveShapeModel) method, or G-EBGM (Gabor feature-based elastic map matching), etc.; the realization of face alignment can include but not limited to: AAM (ActiveAppearance Model) , CLM (ConstrainedLocalModels, constrained local model), etc.

In addition, the key points of the human face can be conventional key points, such as 68 key points of the human face, but not limited thereto. In the embodiment of the present invention, 21 key points of the human face can be used, and these 21 key points include respectively: 3 key points on each side of the eyebrow (brow head, eyebrow tail, and eyebrow peak), 3 key points on each side of the eye (inner corner, outer corner, pupil center), 4 key points on the nose (both sides of the nose) The outermost point, the tip of the nose, the lowermost point of the tip of the nose), 5 key points of the mouth (two lip corners, the upper lip depression point, the lower lip depression point, the middle point of the contact line between the lower lip and upper lip). On the one hand, these 21 key points represent the key parts of the human face, which can effectively represent the features of the human face; on the other hand, the training of the convolutional neural network model of the embodiment of the present invention can be completed through these 21 key points, reducing The amount of data and the cost of training.

Step S404: Extracting facial expression features from the sample image through the convolutional layer of the CNN model to obtain a facial expression feature map.

In this embodiment, the convolution part of the CNN model can adopt the convolution structure of the conventional CNN model, and the processing of the sample image can refer to the processing of the convolution layer part of the relevant CNN model, which will not be repeated here. After being processed by the convolutional layer part, the corresponding facial expression feature map is obtained (the processing result of a certain convolutional layer part can be understood as the output result of the CNN model during a certain training process).

Step S406: Determine the ROI corresponding to each facial key point in the facial expression feature map.

The facial expression feature map output by the convolutional layer contains the processing results of the entire image. If this result is directly used for subsequent expression training, on the one hand, the amount of data to be processed is large, and on the other hand, it cannot be targeted at human Targeted training for facial expressions, resulting in inaccurate training results.

For this reason, in the solution of the embodiment of the present invention, when determining the ROI corresponding to each face key point according to the information of the face key points, in the facial expression feature map, determine according to the coordinates of each face key point Corresponding to each position; with each determined position as a reference point (such as the center point, in practical applications, the center point is allowed to have a small deviation in the range), obtain the area of the set range, and determine each area obtained is the corresponding ROI. Take the 21 key points of the face as an example. When determining the ROI, you can first map it back to the face output by the last convolutional layer of CNN according to the key points of the face, that is, the coordinates of the 21 key points of the face. On the expression feature map, then, with each key point on the face expression feature map as the center, cut out a certain range of areas (generally the cut range is 3 × 3 ~ 7 × 7, preferably 3 × 3), with this The feature maps of 21 regions are used as input to the ROI Pooling Layer (ROI Pooling Layer). These 21 areas cover all positions related to facial expressions, and there is no redundant information, so that the network can focus more on learning these areas, and it is easier to capture subtle changes in facial muscles.

Step S408: performing pooling processing on each of the determined ROIs through the pooling layer part of the CNN to obtain a pooled ROI feature map.

In the CNN model, the pooling layer is often behind the convolutional layer, and the feature vector output by the convolutional layer is reduced by pooling, and the result is improved at the same time, so that the result is not prone to overfitting. For different images, the size and stride of the pooling window can be dynamically calculated according to the size of the image to obtain pooling results of the same size.

In the embodiment of the present invention, the ROI is input into the pooling layer, and after the ROI pooling processing of the pooling layer, a feature representation of the ROI with a fixed length, that is, a ROI feature map with a uniform size can be obtained.

Step S410: Adjust network parameters of the CNN model at least according to the pooled ROI feature map.

In a feasible way, the ROI feature map of the set size after ROI pooling can be input into the fully connected layer for corresponding processing; then, the processed ROI of the set size can be input into the loss layer to obtain a pair of sample images The expression classification result error of expression classification, for example, the error can be the output result of the loss layer; according to the expression classification result error, adjust the network parameters of the CNN model.

After obtaining the processing result of the ROI pooling layer, the processing result can be input into the fully connected layer, and images of different sizes are converted into features of the same dimension through the fully connected layer; then, the features output by the fully connected layer are input into the loss layer, Obtain a loss result; decide whether to adjust the network parameters of the CNN model to continue training according to the loss result. Specifically in this embodiment, after obtaining the ROI feature map of the ROI pooling layer, the ROI feature map is input into the fully connected layer to obtain the ROI feature of the set dimension, wherein the set dimension can be determined by those skilled in the art according to actual needs Appropriately set, the embodiment of the present invention does not limit this; the ROI feature of the set dimension is input into the loss layer, and the loss result is calculated through the loss function; then judge whether the training output of the CNN model meets the convergence condition according to the loss result; if the convergence condition is satisfied , then end the training of the CNN model; if the convergence condition is not met, adjust the parameters of the CNN model training (including but not limited to weight parameter weight, bias parameter bias, etc.) according to the loss result; use the adjusted parameters to continue CNN The model is trained until the training result meets the convergence condition. Wherein, the convergence condition may be appropriately set by those skilled in the art according to actual needs, which is not limited in this embodiment of the present invention.

In this embodiment, the loss function of the loss layer adopts the logistic regression loss function. In this case, the ROI of the set size output by the fully connected layer will be input into the loss layer, and the expression classification result error is performed through the logistic regression loss function of the loss layer. Calculate, and then enter the error calculation result. Optionally, the logistic regression loss function is a logistic regression loss function with a set number of expression classifications. For example, ten types of expressions are marked in the sample image, and the training purpose of the CNN model is to be able to recognize and classify these ten types of expressions, then the logistic regression loss function can be a ten-category logistic regression loss function, where, "Ten categories" means that the logistic regression loss function can detect and recognize the ten types of marked expressions.

In addition, in the actual CNN model training, the sample image used for training can also use the facial expression sample image of the video frame sequence. There is a certain connection between the video frames in the video frame sequence, and the video frame sequence is used as the training sample. It is more helpful for the trained CNN model to recognize facial expressions in continuous video frames in actual detection.

Through the above process, the training of the facial expression recognition of the CNN model in the embodiment of the present invention is realized. Different from the conventional training method, after the facial expression sample image is processed by the convolution layer part in the CNN model, according to the human face The key point is to determine the ROI on the processing result of the convolutional layer, input the ROI into the ROI pooling layer for processing, and finally determine the training of the convolutional neural network model according to the processing result of the ROI pooling layer. By selecting the area corresponding to the key points of the face as the ROI, more targeted training can be carried out, and the detailed information of the changes in multiple areas of the face can be fully utilized, and it is more accurate for subtle expression changes and faces of different poses. identification. This CNN model training method using the facial area corresponding to the key points of the face as the ROI can effectively capture subtle expression changes, and at the same time can better handle the differences caused by different facial poses, thereby improving the prediction of the CNN model. precision and robustness. Moreover, compared with the traditional machine learning framework for facial expression recognition, the CNN using the structure of the embodiment of the present invention can not only use a large amount of data samples for training due to its own structural characteristics, but also has high training efficiency and relatively low training cost .

The method for training a convolutional neural network model in this embodiment may be executed by any appropriate device with data processing capabilities, including but not limited to: mobile terminals, PCs, and the like.

Embodiment five

Referring to FIG. 5 , it shows a structural block diagram of an expression recognition device according to Embodiment 5 of the present invention; specifically, it includes the following modules:

The first determination module 502 is used to perform facial expression feature extraction on the human face image to be detected through the convolutional layer part of the convolutional neural network model and the acquired facial key points in the human face image to be detected, and obtain the human face Facial expression feature map.

The second determination module 504 is configured to determine ROIs in the facial expression feature map corresponding to each key point of the human face.

The third determination module 506 is configured to perform pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model, and obtain a pooled ROI feature map.

The fourth determining module 508 is configured to acquire an expression recognition result of the face image at least according to the ROI feature map.

Optionally, the face image includes a static face image.

Optionally, the face image includes a face image in a sequence of video frames.

Optionally, the third determining module 506 includes: a first acquiring submodule 5062, configured to acquire a preliminary image of the face image of the current frame according to the ROI feature map of the face image of the current frame Expression recognition result; the second acquisition submodule 5064 is used to obtain the expression recognition of the face image of the current frame according to the preliminary expression recognition result of the current frame and the expression recognition result of the face image of at least one previous frame result.

Optionally, the second acquisition sub-module 5064 is configured to perform weighting processing on the preliminary facial expression recognition result of the face image of the current frame and the facial expression recognition result of at least one previous frame of the face image , obtaining the expression recognition result of the face image of the current frame, wherein the weight of the preliminary expression recognition result of the face image of the current frame is greater than the weight of the expression recognition result of any previous frame of the face image.

Optionally, the apparatus further includes: a fifth determining module 510, configured to determine that the position of the current frame in the video frame sequence is greater than or equal to a set position threshold.

Optionally, the device further includes: a response module 512, configured to output the face of the face image of the current frame in response to the position of the current frame in the sequence of video frames being smaller than a set position threshold The facial expression recognition result, and/or, saving the facial expression recognition result of the human face image of the current frame.

Optionally, the first determination module 502 is configured to perform face key point detection on the face image to be detected, and obtain face key points in the face image; according to the face key points, through the The convolutional layer part of the convolutional neural network model performs facial expression feature extraction on the face image to obtain a facial expression feature map.

Optionally, the first determination module 502 is configured to extract face key points from the face image to be detected through the convolution layer part of the convolutional neural network model; according to the extracted face key points, the face key points to be detected Extract the facial expression feature from the face image, and obtain the facial expression feature map.

Optionally, the device further includes: a training module 514, configured to acquire a sample image for training, and use the sample image to train the convolutional neural network model, wherein the sample image contains key points of a human face Labeling information of information and facial expressions.

Optionally, the training module 514 includes: a first sub-module 5142, configured to obtain a sample image for training, and perform facial expression feature extraction on the sample image through the convolutional layer part of the convolutional neural network model, Obtain the facial expression feature map; the second submodule 5144 is used to determine the ROI corresponding to each key point of the face in the facial expression feature map; the third submodule 5146 is used to pass the convolution neural The pooling layer part of the network model performs pooling processing on each determined ROI to obtain a pooled ROI feature map; the fourth sub-module 5148 is used to adjust the convolutional neural network model at least according to the ROI feature map network parameters.

Optionally, the second submodule 5142 is configured to determine corresponding positions in the facial expression feature map according to the coordinates of each key point of the face; using the determined positions as reference points, obtain the corresponding Each region of the set range is determined as the corresponding ROI.

Optionally, the third sub-module 5146 is configured to perform pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model to obtain a pooled ROI feature map of a set size; the The fourth sub-module 5148 is used to input the ROI of the set size into the loss layer, and obtain the expression classification result error of the expression classification of the sample image; adjust the convolutional neural network according to the expression classification result error The network parameters of the model.

Optionally, the fourth sub-module 5148 is configured to input the ROI of the set size into the loss layer, calculate and output the expression classification result error through the logistic regression loss function of the loss layer.

Optionally, the logistic regression loss function is a logistic regression loss function with a set number of expression classifications.

Optionally, the sample images of human facial expressions to be trained are sample images of video frame sequences.

Optionally, the sixth determination module 516 is configured to detect the sample images used for training to obtain information about key points of human faces.

The expression recognition device of this embodiment can execute any one of the expression recognition methods in Embodiments 1 to 3, and obtain beneficial effects of the method, which will not be repeated here.

Embodiment six

Referring to FIG. 6 , it shows a structural block diagram of a convolutional neural network model training device according to Embodiment 6 of the present invention; specifically, it includes the following modules:

The first acquiring module 602 is configured to acquire sample images for training and information of corresponding facial key points, wherein the sample images include annotation information of human facial expressions.

The second acquisition module 604 is configured to extract facial expression features from the sample image through the convolutional layer part of the convolutional neural network model to obtain a facial expression feature map.

The third acquiring module 606 is configured to determine ROIs in the facial expression feature map corresponding to each key point of the human face.

The fourth obtaining module 608 is configured to perform pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model, and obtain a pooled ROI feature map.

The fifth obtaining module 610 is configured to adjust network parameters of the convolutional neural network model at least according to the ROI feature map.

Optionally, the third acquisition module 606 is configured to determine corresponding positions in the facial expression feature map according to the coordinates of each key point of the face; using the determined positions as reference points, acquire the corresponding Each region of the set range is determined as the corresponding ROI.

Optionally, the fourth acquisition module 608 is configured to perform pooling processing on each determined ROI through the pooling layer part of the convolutional neural network model, and obtain a pooled ROI feature map of a set size; the The fifth acquisition module 610 includes: a first acquisition submodule 6102, configured to input the ROI of the set size into the loss layer, and acquire an expression classification result error for the expression classification of the sample image; an adjustment submodule 6104, using Adjusting the network parameters of the convolutional neural network model according to the error of the expression classification result.

Optionally, the first obtaining sub-module 6104 is configured to input the ROI of the set size into a loss layer, and calculate and output the expression classification result error through a logistic regression loss function of the loss layer.

Optionally, the logistic regression loss function is a logistic regression loss function with a set number of expression classifications.

Optionally, the sample images of human facial expressions to be trained are sample images of video frame sequences.

Optionally, the device further includes: a sixth acquiring module 612, configured to detect the sample images used for training to obtain information about key points of human faces.

The expression recognition device of this embodiment can execute any one of the expression recognition methods in Embodiments 1 to 3, and obtain beneficial effects of the method, which will not be repeated here.

Embodiment seven

Embodiment 5 of the present invention provides an electronic device, which may be, for example, a mobile terminal, a personal computer (PC), a tablet computer, a server, and the like. Referring to FIG. 7 below, it shows a schematic structural diagram of an electronic device 700 suitable for implementing a terminal device or a server according to an embodiment of the present invention: as shown in FIG. 7 , the electronic device 700 includes one or more processors, communication elements etc., the one or more processors are for example: one or more central processing units (CPU) 701, and/or one or more image processors (GPU) 713, etc., the processors can be stored in the read-only memory ( ROM) 702 or loaded from the storage section 708 into random access memory (RAM) 703 to execute various appropriate actions and processes. Communication elements include a communication component 712 and/or a communication interface 709 . Wherein, the communication component 712 may include but not limited to a network card, and the network card may include but not limited to an IB (Infiniband) network card, and the communication interface 709 includes a communication interface of a network interface card such as a LAN card, a modem, etc. The network performs communication processing.

The processor can communicate with the read-only memory 702 and/or the random access memory 703 to execute executable instructions, connect to the communication component 712 through the communication bus 704, and communicate with other target devices through the communication component 712, thereby completing the embodiment of the present invention The operation corresponding to any of the expression recognition methods provided, for example, through the convolutional layer part of the convolutional neural network model and the key points of the face in the face image to be detected, perform face recognition on the face image to be detected The expression feature is extracted to obtain the facial expression feature map; determine the region of interest ROI corresponding to each face key point in the facial expression feature map; each ROI determined by the pooling layer part of the convolutional neural network model Perform pooling processing to obtain a pooled ROI feature map; at least obtain an expression recognition result of the face image based on the ROI feature map.

In addition, in the RAM 703, various programs and data necessary for device operation can also be stored. The CPU 701 or GPU 713 , the ROM 702 , and the RAM 703 are connected to each other through a communication bus 704 . In the case of RAM703, ROM702 is an optional module. RAM703 stores executable instructions, or writes executable instructions into ROM702 during operation, and the executable instructions cause the processor to perform operations corresponding to the above communication methods. An input/output (I/O) interface 705 is also connected to the communication bus 704 . The communication component 712 can be integrated, or can be configured to have multiple sub-modules (such as multiple IB network cards), and be linked on a communication bus.

The following components are connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 708 including a hard disk, etc. and a communication interface 709 including a network interface card such as a LAN card, a modem, or the like. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc. is mounted on the drive 710 as necessary so that a computer program read therefrom is installed into the storage section 708 as necessary.

It should be noted that the architecture shown in Figure 7 is only an optional implementation, and in the actual practice process, the number and type of components in Figure 7 above can be selected, deleted, added or replaced according to actual needs; Different functional components can also be set separately or integrated. For example, the GPU and CPU can be set separately or the GPU can be integrated on the CPU. The communication components can be set separately or integrated on the CPU or GPU. and many more. These alternative implementations all fall within the protection scope of the present invention.

In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, the computer program includes program code for executing the method shown in the flow chart, and the program code may include corresponding execution The instructions corresponding to the method steps provided by the embodiments of the present invention, for example, use the convolution layer part of the convolutional neural network model and the key points of the face in the face image to be detected to perform face recognition on the face image to be detected. The expression feature is extracted to obtain the facial expression feature map; determine the region of interest ROI corresponding to each face key point in the facial expression feature map; each ROI determined by the pooling layer part of the convolutional neural network model Perform pooling processing to obtain a pooled ROI feature map; at least obtain an expression recognition result of the face image based on the ROI feature map. In such an embodiment, the computer program may be downloaded and installed from a network via a communication element, and/or installed from a removable medium 711 . When the computer program is executed by the processor, the above-mentioned functions defined in the methods of the embodiments of the present invention are executed.

Embodiment Eight

Embodiment 8 of the present invention provides an electronic device, which may be, for example, a mobile terminal, a personal computer (PC), a tablet computer, a server, and the like. Referring to FIG. 8 below, it shows a schematic structural diagram of an electronic device 800 suitable for implementing a terminal device or a server according to an embodiment of the present invention: as shown in FIG. 8 , the electronic device 800 includes one or more processors, communication elements etc., the one or more processors are for example: one or more central processing units (CPU) 801, and/or one or more image processors (GPU) 813, etc., the processors can be stored in the read-only memory ( ROM) 802 or executable instructions loaded from storage section 808 into Random Access Memory (RAM) 803 to perform various appropriate actions and processes. The communication elements include communication component 812 and/or communication interface 809 . Wherein, the communication component 812 may include but not limited to a network card, and the network card may include but not limited to an IB (Infiniband) network card, and the communication interface 809 includes a communication interface of a network interface card such as a LAN card, a modem, etc. The network performs communication processing.

The processor can communicate with the read-only memory 802 and/or the random access memory 803 to execute executable instructions, connect to the communication component 812 through the communication bus 804, and communicate with other target devices through the communication component 812, thereby completing the embodiment of the present invention The operation corresponding to any one of the convolutional neural network model training methods provided, for example, obtaining the sample image for training and the information of the corresponding key points of the human face, wherein the sample image contains the annotation information of human facial expression; by The convolutional layer part of the convolutional neural network model carries out facial expression feature extraction to the sample image to obtain a facial expression feature map; determine the regions of interest corresponding to the key points of each face in the facial expression feature map ROI; pooling each determined ROI through the pooling layer part of the convolutional neural network model to obtain a pooled ROI feature map; adjusting the network of the convolutional neural network model at least according to the ROI feature map parameter.

In addition, in the RAM 803, various programs and data necessary for the operation of the device can also be stored. The CPU 801 or GPU 813 , the ROM 802 , and the RAM 803 are connected to each other through a communication bus 804 . In the case of RAM803, ROM802 is an optional module. RAM803 stores executable instructions, or writes executable instructions into ROM802 during operation, and the executable instructions cause the processor to perform operations corresponding to the above communication methods. An input/output (I/O) interface 805 is also connected to the communication bus 804 . The communication component 812 can be integrated, or can be configured to have multiple sub-modules (such as multiple IB network cards), and be linked on a communication bus.

The following components are connected to the I/O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc. and a communication interface 809 including a network interface card such as a LAN card, a modem, or the like. A drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 810 as necessary so that a computer program read therefrom is installed into the storage section 808 as necessary.

It should be noted that the architecture shown in Figure 8 is only an optional implementation, and in the actual practice process, the number and types of the components in Figure 8 above can be selected, deleted, added or replaced according to actual needs; Different functional components can also be set separately or integrated. For example, the GPU and CPU can be set separately or the GPU can be integrated on the CPU. The communication components can be set separately or integrated on the CPU or GPU. and many more. These alternative implementations all fall within the protection scope of the present invention.

In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, the computer program includes program code for executing the method shown in the flow chart, and the program code may include corresponding execution The instructions corresponding to the method steps provided by the embodiments of the present invention, for example, obtain training sample images and corresponding face key point information, wherein the sample images include labeling information of human facial expressions; through the convolutional neural network The convolutional layer part of the model carries out facial expression feature extraction to described sample image, obtains facial expression feature map; Determines the region of interest ROI corresponding to each face key point respectively in described facial expression feature map; The pooling layer part of the convolutional neural network model performs pooling processing on each determined ROI to obtain a pooled ROI feature map; at least according to the ROI feature map, adjust the network parameters of the convolutional neural network model. In such an embodiment, the computer program may be downloaded and installed from a network via a communication element, and/or installed from a removable medium 811 . When the computer program is executed by the processor, the above-mentioned functions defined in the methods of the embodiments of the present invention are executed.

It should be pointed out that, according to implementation requirements, each component/step described in the embodiment of the present invention can be divided into more components/steps, and two or more components/steps or partial operations of components/steps can also be combined into New components/steps to achieve the purpose of the embodiments of the present invention.

The above method according to the embodiment of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or implemented by Computer code downloaded from a network that is originally stored on a remote recording medium or a non-transitory machine-readable medium and will be stored on a local recording medium so that the methods described herein can be stored on a computer code using a general-purpose computer, a dedicated processor, or a programmable Such software processing on a recording medium of dedicated hardware such as ASIC or FPGA. It will be appreciated that a computer, processor, microprocessor controller, or programmable hardware includes memory components (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when When accessed and executed by a processor or hardware, the processing methods described herein are implemented. Furthermore, when a general-purpose computer accesses the code for implementing the processing shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the processing shown here.

Those skilled in the art can appreciate that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the embodiments of the present invention.

The above embodiments are only used to illustrate the embodiments of the present invention, rather than to limit the embodiments of the present invention. Those of ordinary skill in the relevant technical field can also make various Changes and modifications, so all equivalent technical solutions also belong to the category of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims (10)

1. a kind of expression recognition method, which is characterized in that including：

It is right by the face key point in the convolution layer segment of convolutional neural networks model and the facial image to be detected of acquisition Facial image to be detected carries out human face expression feature extraction, obtains human face expression characteristic pattern；

Determine region of interest ROI corresponding with each face key point in the human face expression characteristic pattern；

Pond processing is carried out to determining each ROI by the pond layer segment of convolutional neural networks model, obtains the ROI behind pond Characteristic pattern；

The Expression Recognition result of the facial image is obtained according at least to the ROI feature figure.

2. according to the method described in claim 1, it is characterized in that, the facial image includes Static Human Face image.

3. according to the method described in claim 1, it is characterized in that, the facial image includes the face figure in sequence of frames of video Picture.

4. according to the method described in claim 3, it is characterized in that, obtain the face figure according at least to the ROI feature figure The Expression Recognition of picture as a result, including：

According to the ROI feature figure of the facial image of the present frame, the preliminary table of the facial image of the present frame is obtained Feelings recognition result；

According to the preliminary Expression Recognition result of the present frame and at least Expression Recognition of the facial image of a prior frame as a result, obtaining Take the Expression Recognition result of the facial image of the present frame.

5. according to the method described in claim 4, it is characterized in that, according to the preliminary Expression Recognition result of the present frame and extremely The Expression Recognition of the facial image of a few prior frame is as a result, obtain the Expression Recognition of the facial image of the present frame as a result, packet It includes：

By the preliminary facial expression recognition result of the facial image of the present frame and the people of at least facial image of a prior frame Face Expression Recognition result is weighted processing, obtains the Expression Recognition of the facial image of the present frame as a result, wherein, described to work as The weight of the preliminary Expression Recognition result of the facial image of previous frame is more than the Expression Recognition result of the facial image of any prior frame Weight.

6. a kind of convolutional neural networks model training method, which is characterized in that including：

The sample image of training and the information of corresponding face key point are obtained, wherein, someone is included in the sample image The markup information of face expression；

Human face expression feature extraction is carried out to the sample image by the convolution layer segment of convolutional neural networks model, obtains people Face expressive features figure；

Determine region of interest ROI corresponding with each face key point in the human face expression characteristic pattern；

Pond processing is carried out to determining each ROI by the pond layer segment of convolutional neural networks model, obtains the ROI behind pond Characteristic pattern；

According at least to the ROI feature figure, the network parameter of the convolutional neural networks model is adjusted.

7. a kind of expression recognition apparatus, which is characterized in that including：

First determining module, for the facial image to be detected for passing through the convolution layer segment of convolutional neural networks model and obtaining In face key point, human face expression feature extraction is carried out to facial image to be detected, obtains human face expression characteristic pattern；

2nd the 5th determining module, for determining sense corresponding with each face key point in the human face expression characteristic pattern Interest region ROI；

Third determining module, the pond layer segment for passing through convolutional neural networks model carry out pond Hua Chu to determining each ROI Reason obtains the ROI feature figure behind pond；

4th determining module, for obtaining the Expression Recognition result of the facial image according at least to the ROI feature figure.

8. a kind of convolutional neural networks model training apparatus, which is characterized in that including：

First acquisition module, for obtaining the information of the sample image of training and corresponding face key point, wherein, the sample Include the markup information of human face expression in this image；

Second acquisition module, the convolution layer segment for passing through convolutional neural networks model carry out face table to the sample image Feelings feature extraction obtains human face expression characteristic pattern；

Third acquisition module, it is corresponding interested with each face key point in the human face expression characteristic pattern for determining Region ROI；

4th acquisition module, the pond layer segment for passing through convolutional neural networks model carry out pond Hua Chu to determining each ROI Reason obtains the ROI feature figure behind pond；

5th acquisition module, for according at least to the ROI feature figure, adjusting the network ginseng of the convolutional neural networks model Number.

9. a kind of electronic equipment, which is characterized in that including：Processor, memory, communication device and communication bus, the processing Device, the memory and the communication device complete mutual communication by the communication bus；

For the memory for storing an at least executable instruction, the executable instruction makes the processor perform right such as will Seek any expression recognition methods of 1-5.

10. a kind of electronic equipment, which is characterized in that including：Processor, memory, communication device and communication bus, the processing Device, the memory and the communication device complete mutual communication by the communication bus；

For the memory for storing an at least executable instruction, the executable instruction makes the processor perform right such as will Seek the 6 convolutional neural networks model training methods.