CN112364736A - Dynamic facial expression recognition method, device and equipment - Google Patents

Abstract

The invention discloses a method, device and equipment for dynamic facial expression recognition. The method comprises: collecting an original video; performing face recognition on an image in the original video, and obtaining a face image position including a human face in the original video; Obtain the face image according to the position of the face image, preprocess the face image to generate the target face image; input the target face image into the trained face expression recognition model, and obtain the recognition result according to the face expression recognition model. Facial expression categories. The dynamic face recognition method of the embodiment of the present invention can realize accurate recognition of facial expressions even in the presence of noise, and has strong robustness and high recognition accuracy.

Description

A kind of dynamic facial expression recognition method, device and equipment

technical field

The present invention relates to the technical field of image processing, and in particular, to a method, device and equipment for dynamic facial expression recognition.

Background technique

With the rise of the artificial intelligence industry, facial expression recognition based on deep learning has attracted more and more attention. Especially in online live classes, by analyzing the facial expressions of students in the live video, we can get the current state of the students attending the class, so as to better Conducive to teacher management and teaching. To recognize static expressions, the usual practice is to input one or more face images in a specific scene, then select a method to recognize the expressions of the faces in the images, and finally output the expression recognition results. When recognizing dynamic expressions, it is necessary to acquire the dynamic facial expression sequences of people as input in real time, and then store and recognize the acquired expression sequences. Therefore, in the process of dynamic expression recognition, it is necessary to ensure the real-time detection of the face, and to maintain the continuous and uninterrupted tracking of the face image. The traditional methods for dynamic expression recognition mainly include geometric method, optical flow method and differential image method. Although these methods have high recognition accuracy, they are more sensitive to noise such as illumination and occlusion, and have poor anti-interference performance.

Therefore, the existing technology still needs to be improved and developed.

SUMMARY OF THE INVENTION

In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a dynamic facial expression recognition method, device and equipment, aiming to solve the technical problem that the dynamic facial expression recognition method in the prior art is more sensitive to noise and has poor anti-interference performance. question.

The technical scheme of the present invention is as follows:

A dynamic facial expression recognition method, the method comprising:

capture original video;

Perform face recognition on the image in the original video, and obtain the position of the face image containing the face in the original video;

Obtain the face image according to the position of the face image, and preprocess the face image to generate the target face image;

Input the target face image into the trained face expression recognition model, and obtain the face expression category according to the recognition result of the face expression recognition model.

Further, the collection of the original video includes:

The camera continuously collects face images to generate original videos.

Further preferably, the described image in the original video is subjected to face recognition, and the position of the face image containing the human face in the original video is obtained, including:

Perform face recognition on the images in the original video through the Haar classifier algorithm, and obtain the position of the face image containing the human face in the original video.

Further preferably, obtaining the face image according to the position of the face image, preprocessing the face image, and generating the target face image, including:

Extract the face image from the original video according to the face position, and perform grayscale processing on the face image to generate a grayscale image;

The size of the grayscale image is normalized to generate the target face image.

Preferably, the inputting the target face image into the trained facial expression recognition model, and before acquiring the facial expression category according to the recognition result of the facial expression recognition model, further includes:

Collect face images through cameras to generate training samples;

Obtain the initial convolutional neural network, train the initial convolutional neural network according to the training samples, and generate a facial expression recognition model.

Further, the described image in the original video is identified by the Haar classifier algorithm, and the position of the face image containing the human face in the original video is obtained, including:

The Haar feature of the image in the original video is obtained, and after the Haar feature is processed by the Adaboost algorithm, the position of the face image containing the face in the original video is obtained.

Further, obtaining the Haar feature of the image in the original video, after the Haar feature is processed by the Adaboost algorithm, obtain the position of the face image containing the face in the original video, including:

Obtain the initial frame of the original video, and extract the Haar features in the frame image frame by frame from the initial frame;

After the extracted Haar features are processed by the Adaboost algorithm, the face image is identified from the frame image, and the position of the face image containing the face in the original video is obtained.

Another embodiment of the present invention provides a dynamic facial expression recognition setting, the device includes:

Video capture module, used to capture original video;

The face image position recognition module is used to perform face recognition on the image in the original video, and obtain the position of the face image containing the face in the original video;

The image preprocessing module is used to obtain the face image according to the position of the face image, and preprocess the face image to generate the target face image;

Input the target face image into the trained face expression recognition model, and obtain the face expression category according to the recognition result of the face expression recognition model.

Another embodiment of the present invention provides a dynamic facial expression recognition device, the device includes at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned dynamic facial expression recognition method.

Another embodiment of the present invention also provides a non-volatile computer-readable storage medium storing computer-executable instructions, the computer-executable instructions being stored by one or more When executed by the processor, the one or more processors can be made to execute the above-mentioned dynamic facial expression recognition method.

Beneficial effects: The dynamic face recognition method of the embodiment of the present invention can realize accurate recognition of facial expressions even in the presence of noise, has strong robustness and high recognition accuracy.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings and embodiments, in which:

1 is a flowchart of a preferred embodiment of a dynamic facial expression recognition method of the present invention;

2 is a schematic diagram of functional modules of a preferred embodiment of a dynamic facial expression recognition device of the present invention;

FIG. 3 is a schematic diagram of the hardware structure of a preferred embodiment of a dynamic facial expression recognition device according to the present invention.

Detailed ways

In order to make the objectives, technical solutions and effects of the present invention clearer and clearer, the present invention will be described in further detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. The embodiments of the present invention are described below with reference to the accompanying drawings.

The embodiment of the present invention provides a dynamic facial expression recognition method. Please refer to FIG. 1 . FIG. 1 is a flowchart of a preferred embodiment of a method for dynamic facial expression recognition according to the present invention. As shown in Figure 1, it includes the steps:

Step S100, collecting original video;

Step S200, performing face recognition on the image in the original video, and obtaining the position of the face image containing the human face in the original video;

Step S300, obtaining a face image according to the position of the face image, preprocessing the face image, and generating a target face image;

Step S400: Input the target face image into the trained facial expression recognition model, and obtain the facial expression category according to the recognition result of the facial expression recognition model.

In specific implementation, the facial expression recognition algorithm of the embodiment of the present invention is used to recognize the expressions of students in the live webcasting class. By recognizing the expressions, the students' listening status can be obtained, which provides convenience for subsequent monitoring of the students' listening effects. .

By collecting the face images of students, using algorithms to detect and locate the position of the faces in the video, in order to make the system maintain a balance between real-time performance and computational complexity, the detected face images are grayed and scaled normalized. Finally, use the pre-trained facial expression recognition model to calculate and output the expression category.

Further, capture the original video, including:

The camera continuously collects face images to generate original videos.

During specific implementation, the camera continuously collects face images as input, and the collected images are original videos.

Further, face recognition is performed on the image in the original video, and the position of the face image containing the human face in the original video is obtained, including:

Perform face recognition on the images in the original video through the Haar classifier algorithm, and obtain the position of the face image containing the human face in the original video.

Further, face recognition is performed on the images in the original video through the Haar classifier algorithm, and the position of the face image containing the human face in the original video is obtained, including:

The Haar feature of the image in the original video is obtained, and after the Haar feature is processed by the Adaboost algorithm, the position of the face image containing the face in the original video is obtained.

Further, the Haar feature of the image in the original video is obtained, and after the Haar feature is processed by the Adaboost algorithm, the position of the face image containing the face in the original video is obtained, including:

Obtain the initial frame of the original video, and extract the Haar features in the frame image frame by frame from the initial frame;

After the extracted Haar features are processed by the Adaboost algorithm, the face image is identified from the frame image, and the position of the face image containing the face in the original video is obtained.

In the specific implementation, the Haar classifier algorithm is used to detect and locate the position of the face in the video. The classifier algorithm mainly uses the Haar-like feature and the Adaboost algorithm, and uses the Viola-Jones as the target detection framework, which can start from the initial frame of the video. Compared with other methods, this method has higher recognition accuracy and stronger robustness, and can quickly detect and locate the position of objects.

Further, obtain a face image according to the position of the face image, preprocess the face image, and generate a target face image, including:

Extract the face image from the original video according to the face position, and perform grayscale processing on the face image to generate a grayscale image;

The size of the grayscale image is normalized to generate the target face image.

During specific implementation, grayscale and scale normalization processing is performed on the detected face image. Because color images contain information such as color, background, and lighting, the influence of these factors can be eliminated through grayscale processing. In addition, the image sizes detected by the face are different. If the image feature dimension is too large, it will increase the amount of calculation, consume more time, and cannot achieve real-time effect; if the image feature dimension is too small, although the amount of calculation will be reduced, The real-time performance is good, but many important facial expression feature information is lost, resulting in a low recognition accuracy. Therefore, the scale normalization technique is used to uniformly convert the size of the image to 48*48 pixels to achieve a balance between the system recognition accuracy and running time.

Further, input the target face image into the trained facial expression recognition model, and before obtaining the facial expression category according to the recognition result of the facial expression recognition model, it also includes:

Collect face images through cameras to generate training samples;

Obtain the initial convolutional neural network, train the initial convolutional neural network according to the training samples, and generate a facial expression recognition model.

During specific implementation, face images are collected by a camera, training samples are generated, and an initial convolutional neural network is constructed. Pre-train the initial convolutional neural network in advance according to the training samples, generate a facial expression recognition model, and save the model parameters, which is convenient for the system to call at any time.

It can be seen from the above method embodiments that an embodiment of the present invention proposes a dynamic facial expression recognition method, which collects a face image through a camera built in a computer and uses it as a test set, and uses the Haar classifier algorithm to detect and locate the face in the video. In order to make the system maintain a balance between the real-time performance and the amount of calculation, the detected face images are grayed and scaled normalized. Finally, the pre-trained facial expression recognition model is used to calculate and Output expression category. The embodiment of the present invention can realize accurate recognition of facial expressions even in the presence of noise, has strong robustness and high recognition accuracy.

It should be noted that the above steps do not necessarily have a certain sequence. Those of ordinary skill in the art can understand from the description of the embodiments of the present invention that in different embodiments, the above steps may have different execution orders. That is, it can be executed in parallel, or it can be executed interchangeably, and so on.

Another embodiment of the present invention provides a dynamic facial expression recognition device. As shown in FIG. 2 , the device 1 includes:

a video capture module 11, used to capture original video;

The face image position recognition module 12 is used to perform face recognition on the image in the original video, and obtain the position of the face image including the face in the original video;

The image preprocessing module 13 is used for acquiring the face image according to the position of the face image, preprocessing the face image, and generating the target face image;

The expression recognition module 14 is configured to input the target face image into the trained facial expression recognition model, and obtain the facial expression category according to the recognition result of the facial expression recognition model.

For specific implementation manners, refer to the method embodiments, which will not be repeated here.

Another embodiment of the present invention provides a dynamic facial expression recognition device. As shown in FIG. 3 , the device 10 includes:

One or more processors 110 and the memory 120 are described by taking one processor 110 as an example in FIG. 3 . The processor 110 and the memory 120 may be connected by a bus or in other ways. In FIG. 3 , the connection by a bus is used as an example.

The processor 110 is used to complete various control logics of the device 10, which can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM ( Acorn RISCMachine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Also, the processor 110 may be any conventional processor, microprocessor or state machine. The processor 110 may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

As a non-volatile computer-readable storage medium, the memory 120 can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as corresponding to the dynamic facial expression recognition method in the embodiment of the present invention. program instructions. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, ie, implements the dynamic facial expression recognition method in the above method embodiments.

The memory 120 may include a storage program area and a storage data area, wherein the storage program area may store an operation device, an application program required for at least one function; the storage data area may store data created according to the use of the device 10 and the like. Additionally, memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 120 may optionally include memory located remotely from processor 110, which may be connected to device 10 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

One or more units are stored in the memory 120, and when executed by one or more processors 110, execute the dynamic facial expression recognition method in any of the above method embodiments, for example, execute the method steps in FIG. 1 described above. S100 to step S400.

Embodiments of the present invention provide a non-volatile computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described The method steps S100 to S400 in FIG. 1 .

As examples, non-volatile storage media can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM may be configured in a variety of formats such as Synchronous RAM (SRAM), Dynamic RAM, (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous DRAM (SLDRAM) And many forms like direct Rambus (Lambas) RAM (DRRAM). The disclosed memory components or memories of the operating environments described herein are intended to include one or more of these and/or any other suitable types of memory.

Another embodiment of the present invention provides a computer program product comprising a computer program stored on a non-volatile computer-readable storage medium, the computer program comprising program instructions that when executed by a processor , causing the processor to execute the dynamic facial expression recognition method in the above method embodiment. For example, the above-described method steps S100 to S400 in FIG. 1 are performed.

The above-described embodiments are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, Alternatively, it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence, or the parts that make contributions to related technologies. The computer software products can exist in computer-readable storage media, such as ROM/RAM, magnetic disks , optical disc, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods of various embodiments or portions of embodiments.

Conditional language such as "could," "could," "may," or "could," among others, is generally intended to convey unless specifically stated otherwise or otherwise understood within the context as used Particular embodiments can include (while other embodiments do not include) particular features, elements, and/or operations. Thus, such conditional language is also generally intended to imply that the features, elements, and/or operations are irrelevant for one or more embodiments. All are required or one or more implementations must include logic to determine, with or without input or prompting, whether these features, elements and/or operations are included or to be performed in any particular implementation.

What has been described in this specification and the accompanying drawings includes examples that can provide a dynamic facial expression recognition method and apparatus. Of course, not every conceivable combination of elements and/or methods has been described for the purpose of describing the various features of the present disclosure, but it will be appreciated that many additional combinations and permutations of the disclosed features are possible. Therefore, it will be apparent that various modifications can be made in the present disclosure without departing from the scope or spirit of the disclosure. In addition, or in the alternative, other embodiments of the present disclosure may be apparent from consideration of this specification and drawings, and from practice of the present disclosure as presented herein. It is intended that the examples presented in this specification and drawings are to be regarded in all respects as illustrative and not restrictive. Although specific terms are employed herein, they are used in a generic and descriptive sense and not for purposes of limitation.

Claims (10)

1. a dynamic facial expression recognition method, is characterized in that, described method comprises: capture original video; Perform face recognition on the image in the original video, and obtain the position of the face image containing the face in the original video; Obtain the face image according to the position of the face image, and preprocess the face image to generate the target face image; Input the target face image into the trained face expression recognition model, and obtain the face expression category according to the recognition result of the face expression recognition model. 2. dynamic facial expression recognition method according to claim 1, is characterized in that, described gathering original video, comprises: The camera continuously collects face images to generate original videos. 3. dynamic facial expression recognition method according to claim 2 is characterized in that, the described image in original video is carried out face recognition, obtains the facial image position that comprises human face in original video, comprises: Perform face recognition on the images in the original video through the Haar classifier algorithm, and obtain the position of the face image containing the human face in the original video. 4. The method for recognizing dynamic facial expressions according to claim 3, characterized in that, obtaining a facial image according to the position of the facial image, preprocessing the facial image, and generating a target facial image, comprising: Extract the face image from the original video according to the face position, and perform grayscale processing on the face image to generate a grayscale image; The size of the grayscale image is normalized to generate the target face image. 5. dynamic facial expression recognition method according to claim 4, is characterized in that, described by the facial expression recognition model that the target facial image input is trained, obtains the facial expression according to the recognition result of the facial expression recognition model Before the emoji category, it also includes: Collect face images through cameras to generate training samples; Obtain the initial convolutional neural network, train the initial convolutional neural network according to the training samples, and generate a facial expression recognition model. 6. dynamic facial expression recognition method according to claim 5, is characterized in that, described by Haar classifier algorithm, image in original video is carried out face recognition, obtains the facial image position that comprises human face in original video, include: The Haar feature of the image in the original video is obtained, and after the Haar feature is processed by the Adaboost algorithm, the position of the face image containing the face in the original video is obtained. 7. dynamic facial expression recognition method according to claim 6, is characterized in that, the Haar characteristic of described obtaining image in original video, after Haar characteristic is processed by Adaboost algorithm, obtain the person that comprises human face in original video Face image locations, including: Obtain the initial frame of the original video, and extract the Haar features in the frame image frame by frame from the initial frame; After the extracted Haar features are processed by the Adaboost algorithm, the face image is identified from the frame image, and the position of the face image containing the face in the original video is obtained. 8. A dynamic facial expression recognition device, wherein the device comprises: Video capture module, used to capture original video; The face image position recognition module is used to perform face recognition on the image in the original video, and obtain the position of the face image containing the face in the original video; The image preprocessing module is used to obtain the face image according to the position of the face image, and preprocess the face image to generate the target face image; The facial expression recognition module is used to input the target face image into the trained facial expression recognition model, and obtain the facial expression category according to the recognition result of the facial expression recognition model. 9. A dynamic facial expression recognition device, wherein the device comprises at least one processor; and, a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the process of any one of claims 1-7 Dynamic facial expression recognition method. 10. A non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, which when executed by one or more processors , which can cause the one or more processors to execute the dynamic facial expression recognition method according to any one of claims 1-7.

Images