CN111325190B - Expression recognition method and device, computer equipment and readable storage medium - Google Patents

Abstract

The invention discloses an expression recognition method, which includes: detecting the position of key points of the face on the face image to obtain the position information of the key points of the face; inputting the face image into four cascaded convolution modules to perform feature processing, Obtain the characteristic response map output by the fourth convolution module; input the characteristic response map into the global average pooling layer module to obtain the feature vector of the first dimension; perform the characteristic response map output by the first three convolution modules Key point feature extraction to obtain key point feature information; connect the feature vector of the first dimension with the key point feature information to obtain the feature vector of the second dimension; input the feature vector of the second dimension to the fully connected layer module In , the feature vector of the third dimension is obtained; the feature vector of the third dimension is input into the trained neural network classifier, and the expression category information of the face image is output. The invention has a simple structure and a small amount of parameters.

Description

A facial expression recognition method, device, computer equipment and readable storage medium

technical field

The invention relates to the technical field of graphics processing. More specifically, it relates to an expression recognition method, device, computer equipment and readable storage medium.

Background technique

Deep learning technology has achieved rapid development. Google, Facebook, Baidu and other companies have invested huge capital and manpower in deep learning technology research, and have continuously launched their unique products and technologies. Other companies such as IBM, Microsoft, Amazon, etc. In the field of deep learning, and has achieved certain results.

Deep learning technology has made breakthroughs in the field of human data perception, such as describing image content, recognizing objects in complex environments in images, and performing speech recognition in noisy environments. At the same time, deep learning technology can also solve image generation and fusion. The problem.

At present, face feature recognition is a hot technology in biological pattern recognition in recent years. This technology requires the detection and positioning of facial feature points, and based on these feature points for face matching, expression analysis and other applications. In recent years, Many research institutions and companies have invested a lot of resources in the field of target recognition, and obtained a series of results. These results have also been used in many industries such as security, finance, life and entertainment. Expression recognition is a facial feature recognition technology. It is also a difficult point in this field. Due to the complexity of human facial expressions, it has been difficult to achieve a substantial breakthrough in the accuracy of machine learning methods for classifying expressions. The development of deep learning provides a basis for the performance improvement of image pattern recognition. There are more possibilities, so the research on expression recognition based on deep learning technology is also a hot focus in the field of facial feature recognition in recent years.

In the prior art, most of the current expression recognition methods use the key points of the face to intercept the face image, enlarge the intercepted eyes and mouth images to the size of the adult face image, and input them into the deep learning network for training together. A deep learning model for expression recognition is obtained, but this method has a complex model structure and a large number of parameters.

Contents of the invention

In order to solve the technical problems proposed in the background technology, the first aspect of the present invention proposes an expression recognition method, comprising the following steps:

Perform face key point position detection on the face image to obtain face key point position information;

The face image is input into four cascaded convolution modules, and the input face images are sequentially subjected to feature processing to obtain a feature response map output by the fourth convolution module;

Inputting the feature response map output by the fourth convolution module into the global average pooling layer module to obtain the feature vector of the first dimension;

Using the key point position information of the human face to carry out key point feature extraction to the feature response graphs output respectively by the first three convolution modules, to obtain the key point feature information of the feature response graphs output respectively by the first three convolution modules;

Connecting the feature vectors of the first dimension with the key point feature information of the feature response graphs respectively output by the first three convolution modules to obtain the feature vectors of the second dimension;

The feature vector of the second dimension is input into the fully connected layer module for processing to obtain the feature vector of the third dimension;

The feature vector of the third dimension is input into the trained neural network classifier, so that the neural network classifier outputs the expression category information of the human face image.

Optionally, the performing detection of key points of the face on the face image to obtain the position information of the key points of the face includes:

Based on the Dlib library, the facial key point position detection is performed on the human face image, and the key points of eyes and mouth in the human face image are obtained as the key point position information of the human face.

Optionally, the convolution module includes: an input layer, a convolution layer, a normalization layer, an activation function layer, a pooling layer, and an output layer;

Wherein, the input end of the convolution layer is connected to the input layer, the input end of the normalization layer is connected to the output end of the convolution layer, and the input end of the activation function layer is connected to the normalization layer. The output end of the pooling layer is connected, the input end of the pooling layer is connected to the output end of the activation function layer, and the input end of the output layer is connected to the output end of the pooling layer.

Optionally, using the key point position information of the human face to perform key point feature extraction on the feature response graphs respectively output by the first three convolution modules, to obtain the feature response graphs respectively output by the first three convolution modules Key point feature information includes:

Using the key point position information of the human face, extract the response value corresponding to the key point position information of the human face in the feature response graph output by each convolution module;

The weighted average of the corresponding response values of the key point position information of the human face in each feature response graph is obtained to obtain the key point feature information of the feature response graphs respectively output by each convolution module.

Optionally, the key point feature information is obtained by the following formula:

为人脸关键点位置信息在特征响应图中第n个通道的响应值，N为特征响应图的通道数量。Among them, K _{i, j} is the key point feature information,

is the response value of the nth channel in the feature response map of the position information of the key points of the face, and N is the number of channels in the feature response map.

Optionally, the step before extracting the response value corresponding to the key point position information of the face from the feature response map output by each convolution module by using the key point position information of the face further includes:

The size of the characteristic response map output by each convolution module is adjusted to be the same as the size of the face image.

Optionally, the steps before the face key point position detection is carried out on the face image to obtain the face key point position information also include:

Get the input image, perform face detection on the input image, and adjust the size of the detected face image to a preset size.

Optionally, the neural network classifier is obtained through stochastic gradient descent training.

The second aspect of the present invention proposes an expression recognition device, comprising:

The face key point position detection module is used to detect the face key point position on the face image to obtain the face key point position information;

Four cascaded convolution modules are used to input the face image, and sequentially perform feature processing on the input face image to obtain a feature response map output by the fourth convolution module;

The global average pooling layer module is used to obtain the feature vector of the first dimension according to the characteristic response map output by the input fourth convolution module;

The key point feature information module is used to use the key point position information of the human face to perform key point feature extraction on the feature response graphs output by the first three convolution modules respectively, and obtain the feature responses output respectively by the first three convolution modules The key point feature information of the graph;

A eigenvector connection module, configured to connect the eigenvectors of the first dimension with the key point feature information of the eigenresponse graphs respectively output by the first three convolution modules to obtain the eigenvectors of the second dimension;

A fully connected layer module, configured to process the input feature vector of the second dimension to obtain a feature vector of the third dimension;

The neural network classifier is used to input the feature vector of the third dimension into the trained neural network classifier, so that the neural network classifier outputs the expression category information of the human face image.

The third aspect of the present invention proposes a computer device, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the program, the computer program described in the first aspect of the present invention is implemented described method.

The fourth aspect of the present invention provides a computer-readable storage medium, the computer-readable storage medium stores instructions, and when the computer-readable storage medium runs on a computer, the computer executes the first method of the present invention. method described in the aspect.

The beneficial effects of the present invention are as follows:

The technical scheme of the present invention has the advantages of clear principle and simple design, and specifically utilizes the mechanism of key point feature extraction from the feature response graph based on the position information of key points of the face, so as to achieve the purpose of performing corresponding expression recognition on the input face image , the structure is simple and the number of parameters is small.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

Fig. 1 shows the flowchart of a kind of facial expression recognition method that an embodiment of the present invention proposes;

Fig. 2 shows the schematic diagram of the algorithm structure of expression recognition method in the present embodiment;

Fig. 3 shows the schematic diagram of the key point position of human face;

FIG. 4 shows a schematic structural diagram of a convolution module in this embodiment;

Fig. 5 shows the flowchart of key point feature extraction for the feature response graphs of the first three convolution modules in this embodiment;

FIG. 6 shows a schematic structural diagram of a computer device proposed by another embodiment of the present invention.

Detailed ways

In order to make the technical solutions and advantages of the present invention clearer, the implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings.

Fig. 1 shows a flow chart of the steps of an expression recognition method proposed by an embodiment of the present invention, the expression recognition method can be applied to a terminal device, and the terminal device can be a smart phone, a tablet computer, a personal computer or a server, etc., for To facilitate understanding, the following briefly introduces the algorithm structure of the facial expression recognition method.

As shown in Figure 2, the algorithm structure of the facial expression recognition method in the present embodiment includes a human face image input layer, a human face key point position detection module, 4 cascaded convolutional layer modules, a global average pooling layer module, Key point feature information module, feature vector connection module, fully connected layer and classifier;

in,

The face key point position detection is used to detect the face key point position on the face image to obtain the face key point position information;

Four cascaded convolution modules are used to input the face image, and feature processing is performed on the input face image in turn to obtain a feature response map output by the fourth convolution module;

The global average pooling layer module is used to obtain the feature vector of the first dimension according to the characteristic response map output by the input fourth convolution module;

The key point feature information module is used to use the key point position information of the face to extract key point features from the feature response maps output by the first three convolution modules respectively, and obtain the feature responses output by the first three convolution modules respectively The key point feature information of the graph;

The feature vector connection module is used to connect the feature vectors of the first dimension with the key point feature information of the feature response graphs respectively output by the first three convolution modules to obtain the feature vectors of the second dimension;

The fully connected layer module is used to process the input feature vector of the second dimension to obtain a feature vector of the third dimension;

The classifier is used to input the feature vector of the third dimension into the trained neural network classifier, so that the neural network classifier can output the expression category information of the human face image.

Here, the facial expression category information may be happy, surprised, calm, sad, angry, disgusted and fearful, and of course, other types of facial expressions may also be preset.

The algorithm structure of the expression recognition method has been introduced above, and the expression recognition method is described in detail below. The expression recognition method is shown in Figure 1, including:

S100. Perform position detection of key points of the face on the face image to obtain position information of key points of the face;

S200. Input the face images into four cascaded convolution modules, and sequentially perform feature processing on the input face images to obtain a feature response map output by the fourth convolution module;

S300. Input the feature response map output by the fourth convolution module into the global average pooling layer module to obtain a feature vector of the first dimension;

S400. Using the key point position information of the face to extract key point features from the feature response graphs respectively output by the first three convolution modules, and obtain key point feature information of the feature response graphs respectively output by the first three convolution modules ;

S500. Connect the feature vector of the first dimension with the key point feature information of the feature response graph respectively output by the first three convolution modules to obtain the feature vector of the second dimension;

S600. Input the feature vector of the second dimension into the fully connected layer module for processing to obtain the feature vector of the third dimension;

S700. Input the feature vector of the third dimension into the trained neural network classifier, so that the neural network classifier outputs the expression category information of the human face image.

Specifically, in S100, it also includes: detecting the positions of key points of the face on the face image based on the Dlib library, and acquiring key points of eyes and mouth in the face image as position information of key points of the face.

It should be noted that the Dlib library is a comprehensive application library of image processing algorithms similar to OpenCV, which belongs to the prior art, and face key point recognition is a kind of highlight function of the library. The face key point position detection of the Dlib library is Developed based on the random forest algorithm in machine learning, it can describe the positions of 68 key points in the face, as shown in Figure 3, including eyebrows, eyes, nose, mouth and jaw, and the calculation speed is fast. In this implementation In the example, in order to make the deep learning network more focused on the expression features, 32 key points of eyes and mouth, which are most related to expressions, are selected from the 68 key points as the key point position information of the face.

Further, in this embodiment, the steps before S100 also include: acquiring an input image, performing face detection on the input image, and adjusting the size of the detected face image to a preset size.

Specifically, the face in the acquired input image can be detected through the Dlib library, and the size of the detected face image can be uniformly changed to a preset size. Here, the specific size of the preset size can be determined by the staff. The actual needs can be set by yourself, which is not limited in this embodiment. Exemplarily, the preset size can be 48×48.

In S200 , as shown in FIG. 4 , the convolution module may specifically include: an input layer, a convolution layer, a normalization layer, an activation function layer, a pooling layer, and an output layer.

Specifically, the input end of the convolution layer is connected to the input layer, the input end of the normalization layer is connected to the output end of the convolution layer, and the input end of the activation function layer is connected to the normalization layer. The output end of the pooling layer is connected, the input end of the pooling layer is connected to the output end of the activation function layer, and the input end of the output layer is connected to the output end of the pooling layer.

In this embodiment, the role of the four cascaded convolution modules is to perform feature extraction on the input feature response maps of different scales, and output the processed feature response maps. In order to facilitate understanding, according to the four convolution modules by Arranging from top to bottom, the four convolution modules are respectively defined as a first convolution module, a second convolution module, a third convolution module, and a fourth convolution module.

In the specific implementation, the face image will first be input into the first convolution module, as shown in Figure 4, the scale of the first convolution module includes a 3×3 convolution kernel and 32 channels, and the face image is passed through the first volume After processing by the product module, a characteristic response map with a scale of 24×24 and a channel number of 32 is obtained. The output of the first convolution module is used as the input of the second convolution module, which includes a 3×3 convolution kernel, 64 channel, input the characteristic response map with 24×24 and 32 channels into the second convolution module for processing, and obtain the characteristic response map with the scale of 12×12 and 64 channels, the output of the second convolution module is used as the The input of the three convolution modules, the third convolution module includes a 3×3 convolution kernel and 128 channels, and the characteristic response map with 12×12 and 64 channels is input to the third convolution module for processing to obtain a scale of 6 ×6, the characteristic response map with 128 channels, the output of the third convolution module is used as the input of the fourth convolution module, the fourth convolution module includes 3×3 convolution kernels, 256 channels, and the 6×6, channel The feature response map with a number of 128 is input to the fourth convolution module for processing to obtain a feature response map with a scale of 3×3 and a channel number of 256.

In S300, the function of the global average pooling layer module is to convert the characteristic response map output by the fourth convolution module into a feature vector of the first dimension by means of averaging. Here, the first dimension is specifically 1×256.

In S400, as shown in Figure 5, the key point feature extraction is performed on the feature response maps output by the first three convolution modules by using the key point feature information module, that is, the first volume The feature response maps output by the convolution module, the second convolution module, and the third convolution module are used to extract key point features, so as to obtain the output of the first convolution module, the second convolution module, and the third convolution module. Keypoint feature information for the eigenresponse plot.

Specifically, S400 includes the following sub-steps:

Using the key point position information of the human face, extract the response value corresponding to the key point position information of the human face in the feature response graph output by each convolution module;

The weighted average of the corresponding response values of the key point position information of the human face in each feature response graph is obtained to obtain the key point feature information of the feature response graphs respectively output by each convolution module.

In this embodiment, according to the location information of the key points of the face obtained in the preceding steps, the features corresponding to the face are extracted from the feature response maps output by the first convolution module, the second convolution module, and the third convolution module. The response value corresponding to the position information of the key point is to extract the response value corresponding to 32 key points of eyes and mouth, and weight the response value of each key point in the characteristic response graph, and finally obtain the is a set of 32 response values corresponding to the characteristic response maps output by each convolution module.

Further, the key point feature information is obtained by the following formula:

为人脸关键点位置信息在特征响应图中第n个通道的响应值，N为特征响应图的通道数量。Among them, K _{i, j} is the key point feature information,

is the response value of the nth channel in the feature response map of the position information of the key points of the face, and N is the number of channels in the feature response map.

In this embodiment, the step before extracting the response value corresponding to the key point position information of the human face from the feature response map output by each convolution module by using the key point position information of the human face is further include:

The size of the characteristic response map output by each convolution module is adjusted to be the same as the size of the face image.

Specifically, the size of the characteristic response map output by each convolution module can be adjusted to be consistent with the size of the input face image through an upsampling operation.

In S500, the feature vectors of the first dimension obtained in S300 are connected to the key point feature information of the characteristic response graphs respectively output by the first three convolution modules to obtain the feature vectors of the second dimension. Here, in After extracting the key point feature information of the feature response map output by the first three convolution modules, three 1×32-dimensional feature vectors are connected with the feature vector of the first dimension to obtain the feature of the second dimension Vector, here, the second dimension is specifically 1×352.

In S600, each element of the output vector of the fully connected layer module is connected with each element of the input vector, which can fuse all the features of the input vector. Therefore, after the fully connected layer module, the global average pooling layer The feature vector of the first dimension output by the module is fused with the key point feature information of the feature response map output by the first three convolution modules to obtain the feature vector of the third dimension, specifically, the input of the fully connected layer module The eigenvector of the second dimension, output the eigenvector of the third dimension, the third dimension is 1×128.

In S700, the confidence degree of each preset expression category can be calculated by inputting the feature vector of the third dimension into the Softmax layer in the trained neural network, wherein the confidence degree can be obtained by the following formula:

Wherein, j is the sequence number of the expression category, x is the input vector of the softmax layer (that is, the feature vector of the third dimension in the present embodiment), w is the network weight parameter, and P (y=j|x) is the value of the Ssoftmax layer When the input vector is x, the corresponding expression category is the confidence level of the jth expression category.

In this embodiment, the expression category corresponding to the face image to be recognized may be determined according to the confidence of each expression category. Specifically, the expression category with the highest confidence may be determined as the expression category corresponding to the human face image.

It should be noted that the neural network classifier in this embodiment can be trained by the stochastic gradient descent method. First, the neural network to be trained and face image samples of various preset expression categories can be obtained, and then a certain amount of face image samples can be obtained each time. A large number of face image samples are preprocessed, and the preprocessed face image samples are input into the neural network for gradient descent iterative training until the preset training conditions are reached, and a trained neural network classifier is obtained. Wherein, the preset training condition may be: the number of iterations reaches the preset number, or the value of the damage function is smaller than the preset value. In this embodiment, cross-entropy may be used as the damage function.

In this embodiment, the preset expression categories may include: happiness, surprise, calm, sadness, anger, disgust and fear, and of course, other numbers and other types of expression categories may also be preset.

In summary, the technical solution of the present invention has the advantages of clear principle and simple design, and specifically utilizes the mechanism of key point feature extraction of the feature response map based on the key point position information of the human face, so as to achieve corresponding The purpose of facial expression recognition, the structure is simple, and the number of parameters is small.

A kind of facial expression recognition device that another embodiment of the present invention proposes, comprises:

The face key point position detection module is used to detect the face key point position on the face image to obtain the face key point position information;

Four cascaded convolution modules are used to input the face image, and sequentially perform feature processing on the input face image to obtain a feature response map output by the fourth convolution module;

The global average pooling layer module is used to obtain the feature vector of the first dimension according to the characteristic response map output by the input fourth convolution module;

The key point feature information module is used to use the key point position information of the human face to perform key point feature extraction on the feature response graphs output by the first three convolution modules respectively, and obtain the feature responses output respectively by the first three convolution modules The key point feature information of the graph;

A eigenvector connection module, configured to connect the eigenvectors of the first dimension with the key point feature information of the eigenresponse graphs respectively output by the first three convolution modules to obtain the eigenvectors of the second dimension;

A fully connected layer module, configured to process the input feature vector of the second dimension to obtain a feature vector of the third dimension;

The neural network classifier is used to input the feature vector of the third dimension into the trained neural network classifier, so that the neural network classifier outputs the expression category information of the human face image.

Yet another embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and operable on the processor. The above expression recognition method is realized when the processor executes the program. As shown in Figure 6, the computer system suitable for realizing the server provided by this embodiment includes a central processing unit (CPU), which can be loaded into a random computer according to a program stored in a read-only memory (ROM) or from a storage part. Various appropriate actions and processes are executed by accessing programs in the memory (RAM). In RAM, various programs and data necessary for the operation of the computer system are also stored. The CPU, ROM, and RAM are connected here via a bus. An input/input (I/O) interface is also connected to the bus.

The following parts are connected to the I/O interface: an input section including a keyboard, a mouse, etc.; an output section including a liquid crystal display (LCD) etc., a speaker, etc.; a storage section including a hard disk, etc.; and a network including a LAN card, a modem, etc. The communication part of the interface card. The communication section performs communication processing via a network such as the Internet. Drives are also connected to the I/O interface as required. A removable medium, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive as necessary so that a computer program read therefrom is installed into the storage section as necessary.

In particular, according to this embodiment, the process described in the flow chart above can be implemented as a computer software program. For example, the present embodiment includes a computer program product, which includes a computer program tangibly embodied on a computer-readable medium, the computer program including program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication part, and/or installed from a removable medium.

The flowcharts and schematic diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the system, method and computer program product of the present embodiment. In this regard, each block in a flowchart or schematic diagram may represent a module, program segment, or part of code that includes one or more programmable components for implementing specified logical functions. Execute instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the illustrations and/or flowchart illustrations, and combinations of blocks in the illustrations and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the description in this embodiment can be realized by software or by hardware. The described unit can also be set in a processor, for example, it can be described as: a processor includes a face key point position detection module, four cascaded convolution modules, a global average pooling layer module, and the like. Wherein, the names of these units do not constitute a limitation of the unit itself under certain circumstances.

As another aspect, the present application also provides a computer-readable storage medium, which may be the computer-readable storage medium contained in the device described in the above-mentioned embodiments; A computer-readable storage medium assembled in a terminal. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the facial expression recognition method described in the present invention.

Apparently, the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those of ordinary skill in the art can also make It is not possible to exhaustively list all the embodiments here, and any obvious changes or changes derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims (11)

1. An expression recognition method is characterized by comprising the following steps:

detecting key point positions of the face image to obtain the position information of the key point positions of the face;

inputting the face image into four cascaded convolution modules, and sequentially carrying out feature processing on the input face image to obtain a feature response diagram output by a fourth convolution module;

inputting the characteristic response diagram output by the fourth convolution module into a global average pooling layer module to obtain a characteristic vector of a first dimension number;

extracting key point characteristics of the characteristic response graphs respectively output by the first three convolution modules by utilizing the key point position information of the human face to obtain key point characteristic information of the characteristic response graphs respectively output by the first three convolution modules;

connecting the feature vector of the first dimension with the key point feature information of the feature response graphs respectively output by the first three convolution modules to obtain a feature vector of a second dimension;

inputting the feature vector of the second dimension into a full-connection layer module for processing to obtain a feature vector of a third dimension;

and inputting the feature vector of the third dimension into a trained neural network classifier so as to output expression category information of the face image by the neural network classifier.

2. The expression recognition method of claim 1, wherein,

the step of detecting the key point positions of the human face image to obtain the key point position information of the human face comprises the following steps:

and detecting key points of the face image based on the Dlib library, and acquiring key points of eyes and mouths in the face image as the position information of the key points of the face.

3. The expression recognition method of claim 1, wherein the convolution module comprises: an input layer, a convolution layer, a normalization layer, an activation function layer, a pooling layer and an output layer;

the input end of the convolution layer is connected with the input layer, the input end of the normalization layer is connected with the output end of the convolution layer, the input end of the activation function layer is connected with the output end of the normalization layer, the input end of the pooling layer is connected with the output end of the activation function layer, and the input end of the output layer is connected with the output end of the pooling layer.

4. The expression recognition method of claim 1, wherein,

the step of extracting key point features of the feature response graphs respectively output by the first three convolution modules by utilizing the key point position information of the face comprises the following steps:

extracting response values corresponding to the position information of the key points of the human face from the characteristic response graphs output by each convolution module by utilizing the position information of the key points of the human face;

and carrying out weighted average on response values corresponding to the facial key point position information in each characteristic response graph to obtain key point characteristic information of the characteristic response graph respectively output by each convolution module.

5. The expression recognition method of claim 4, wherein,

the key point characteristic information is obtained by the following formula:

wherein K is _i，j As the feature information of the key points,

and the response value of the nth channel in the characteristic response diagram for the position information of the key points of the human face is obtained, and N is the number of channels of the characteristic response diagram.

6. The expression recognition method of claim 4, wherein,

the step before extracting the response value corresponding to the face key point position information from the feature response graph output by each convolution module by utilizing the face key point position information further comprises the following steps:

and adjusting the size of the characteristic response graph output by each convolution module to be the same as the size of the face image.

7. The expression recognition method of claim 1, wherein,

the step before the step of detecting the key point position information of the human face to the key point position information of the human face further comprises the following steps:

and acquiring an input image, performing face detection on the input image, and adjusting the size of the detected face image to a preset size.

8. The expression recognition method of any one of claims 1 to 7,

the neural network classifier is obtained through training by a random gradient descent method.

9. An expression recognition apparatus, characterized by comprising:

the face key point detection module is used for detecting the face key points of the face image to obtain the position information of the face key points;

the four cascade convolution modules are used for inputting the face images, and sequentially carrying out feature processing on the input face images to obtain a feature response diagram output by the fourth convolution module;

the global average pooling layer module is used for obtaining the feature vector of the first dimension according to the feature response diagram output by the fourth convolution module;

the key point feature information module is used for extracting key point features of the feature response graphs respectively output by the first three convolution modules by utilizing the face key point position information to obtain key point feature information of the feature response graphs respectively output by the first three convolution modules;

the feature vector connection module is used for connecting the feature vector of the first dimension with the key point feature information of the feature response graphs respectively output by the first three convolution modules to obtain the feature vector of the second dimension;

the full-connection layer module is used for processing the input feature vector of the second dimension to obtain a feature vector of a third dimension;

and the neural network classifier is used for inputting the feature vector with the third dimension into the trained neural network classifier so as to output the expression category information of the face image by the neural network classifier.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1-8 when the program is executed by the processor.

11. A computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the method of any of claims 1-8.

Images