CN107194559A - A kind of work stream recognition method based on Three dimensional convolution neutral net - Google Patents

Abstract

The invention discloses a workflow recognition method based on a three-dimensional convolutional neural network. Only dividing different process tasks in advance and manually labeling different actions during the process of analyzing the video does not meet the automation requirements of intelligent manufacturing. The present invention firstly proposes an inter-frame difference method with an adaptive threshold, which is mainly used to segment the region of moving objects from complex backgrounds, thereby reducing the time complexity of subsequent feature extraction and model training; secondly, The 3D convolutional neural network is improved so that it can fully adapt to the factory environment with multiple monitoring devices, and for different views, the view pooling layer is used to fuse the views from different angles according to the weight; finally, a The new action division method automatically divides the continuous production actions in the video, thus realizing the automatic workflow recognition process.

Description

A Workflow Recognition Method Based on 3D Convolutional Neural Network

technical field

The invention belongs to the technical field of workflow identification and is used for fast and accurate identification and detection of production and manufacturing processes. Through the camera installed in the manufacturing workshop, the entire process of production scheduling on the production line is photographed, and then the video is calculated and processed, so as to protect the personal safety of employees, reduce production costs, ensure product quality, and optimize production scheduling and process specifications. play an important role

Background technique

Intelligent manufacturing is the further development direction of manufacturing automation. It widely applies artificial intelligence technology to engineering design, process design, production scheduling, fault diagnosis and other links of industrial manufacturing process, so as to realize intelligent manufacturing process and greatly improve productivity. . As an important technical direction of intelligent manufacturing, workflow recognition has attracted the attention of the industry and scientific research circles. It uses the camera installed in the manufacturing workshop to capture the entire process of production scheduling on the production line, and then calculates and processes the video to realize fast and accurate identification and detection of industrial production processes, thereby protecting the personal safety of employees and reducing production costs. , ensuring product quality, and optimizing production scheduling and process specification will play an important role.

However, workflow identification technology has its complexity and particularity. First of all, because there are many kinds of machines, transport vehicles, auxiliary equipment and other objects in the production workshop, they are often blocked by each other, and the similarity of different process operations, frequent changes in light intensity in the workshop, all these make video and image analysis and Identification poses challenges. In addition, the dynamic production workflow process makes the recognition process quite complicated, and it is easy to produce deviations: for example, different tasks in the workflow often have different execution times, and there is no clear definition between the start and end of tasks; these Tasks may even contain both human and machine actions, some of which have nothing to do with workflow and must be distinguished from real production tasks. These aspects make it difficult for traditional action/posture recognition methods that rely on object detection and tracking to be applied in complex factory manufacturing environments. In addition, although some researchers have carried out some research on workflow recognition technology, how to automatically divide the production process/action of the image sequence in the video, these research works have not given a clear definition, most of them are only Dividing different process tasks in advance and artificially labeling different actions during the video analysis process obviously does not meet the automation requirements of intelligent manufacturing.

Contents of the invention

Aiming at the current research status, the present invention proposes a workflow identification framework with strong robustness. In this framework, an inter-frame difference method with an adaptive threshold is first proposed, which is mainly used to segment the region of moving objects from complex backgrounds, thereby reducing the time complexity of subsequent feature extraction and model training ;Secondly, the 3D convolutional neural network is improved so that it can fully adapt to the factory environment with multiple monitoring devices, and for different views, the view pooling layer is used to fuse the views from different angles according to the weight; finally, the proposed A new action division method is proposed to automatically divide the continuous production actions in the video, so as to realize the automatic workflow recognition process.

The concrete steps of the inventive method are:

Step (1), export workflow videos containing multiple perspectives from the data set, and obtain the video resolution and frame number of the workflow videos of each perspective;

Step (2), initializing the inter-frame difference thresholds of the workflow videos of each perspective; performing steps (3) to (11) for the workflow videos of each perspective;

Step (3), setting t=2;

Step (4), read three consecutive video frames of t-1, t, and t+1 and carry out grayscale and median filter processing of these three video frames;

Step (5), performing inter-frame difference operations on the first two frames and the last two frames respectively to obtain two inter-frame difference images;

Step (6), dynamically update the inter-frame difference threshold according to the two inter-frame difference images obtained in step (5); the inter-frame difference threshold dynamic update method is as follows:

6.1 Set l=1, the inter-frame difference threshold of the tth frame d _k is the pixel value of the kth pixel in the inter-frame difference image, max{d _k } is the maximum value of the pixel value in the inter-frame difference image, and min{d _k } is the minimum value of the pixel value in the inter-frame difference image;

6.2 orders N ₁ and N ₂ respectively represent the satisfaction of with The total number of pixels;

6.3 If then will Assign value to τ ¹ _t , otherwise, let l=l+1, repeat step 6.2;

In step (7), the current frame is binarized according to the inter-frame difference threshold obtained in (6), and the pixels greater than the inter-frame difference threshold are set to 1, and the pixels less than the inter-frame difference threshold are set to 0;

Step (8), running and operating the front and rear two frame difference images to obtain three frame difference images, and using the block extraction method to obtain the central coordinates of the point of interest;

Step (9), segmenting the extracted point of interest from the original image of the current frame;

Step (10), the value of t is gradually increased by 1, and steps (4) to (9) are repeated until the value of t is 1 smaller than the value of the last frame of the workflow video, and the segmentation size of step (9) remains unchanged during the repetition process ; The point-of-interest image obtained in step (9) in each repetition process is preserved as the point-of-interest video in sequence, and the point-of-interest video is classified according to the classification rules in the data set;

Step (11), randomly select 90% from the point-of-interest video obtained in step (10) as a training set, and the rest as a test set;

Step (12), construct a multi-view three-dimensional convolutional neural network, and initialize the number of training rounds to be 5000; the construction method of the multi-view three-dimensional convolutional neural network is as follows:

12.1. The convolution and pooling operations are as follows:

① Initialize a four-dimensional convolution kernel with a size of 9*9*9*10 for the first convolutional layer, the activation function is sigmoid, the window size of the first pooling layer is 2, and the step size is 2;

②Initialize a four-dimensional convolution kernel with a size of 9*9*7*30 for the second convolutional layer, the activation function is sigmoid, the window size of the second pooling layer is 2, and the step size is 2;

③Initialize a four-dimensional convolution kernel with a size of 9*8*5*50 for the third convolutional layer, the activation function is sigmoid, the window size of the third pooling layer is 2, and the step size is 2;

④ Initialize a four-dimensional convolution kernel with a size of 4*3*3*150 for the fourth convolutional layer, the activation function is sigmoid, the window size of the fourth pooling layer is 2, and the step size is 2;

12.2. Initialize the weight parameters of each feature map in the weighted average view pooling layer is a random value in [0,1], and The weighted average view pooling operation in the weighted average view pooling layer is as follows:

In the formula, a is the weighted average feature map after the weighted average view pooling operation, t ₁ is the serial number of the pooled feature map after the convolution and pooling operation, is the weight of the pooled feature map corresponding to the serial number t ₁ , exp represents the exponential function with e as the base, is the pooled feature map corresponding to the serial number t ₁ ;

12.3. Initialize a convolution kernel of 3000*1500 and 1500*750 for the first two fully connected layers, and set the activation function to Relu; the weighted average feature map after the weighted average view pooling operation is input to the first two fully connected layers ;

12.4. Initialize a 750*14 convolution kernel for the last fully connected layer and set the Softmax classification function.

In step (13), 20 videos are randomly selected from the training set corresponding to the workflow videos of each perspective and input into the multi-view 3D convolutional neural network in (12) for feature training, and the training error is output;

Step (14), randomly select 10 videos from the training set corresponding to the workflow videos of each perspective and input them into the multi-view three-dimensional convolutional neural network for verification, and obtain the accuracy rate of classification and recognition of the multi-view three-dimensional convolutional neural network;

Step (15), repeating steps (13) to (14), the number of training rounds is reduced by 1 each time it is repeated, until the number of training rounds is 0, and a trained multi-view three-dimensional convolutional neural network is obtained;

Step (16), testing the multi-view three-dimensional convolutional neural network in step (15) using the test set corresponding to the workflow video of each perspective;

Step (17), for the newly input workflow video, obtain the video resolution and the number of frames, and initialize the difference threshold between frames; set t=2;

Step (18), according to steps (4) to (8), extract the center coordinates of two adjacent frames of interest points, and calculate the distance between the two center coordinates, if the distance is greater than the set threshold T, mark it as a motion state S ₁ , otherwise, it is marked as relative static state S ₀ ;

Step (19), the value of t is gradually increased by 1, and step (18) is repeated until the value of t is smaller than the value of the last frame of the newly input workflow video by 1, and the number of continuous S ₀ and S ₁ is counted. When the number of S ₀ or S ₁ is greater than or equal to N, the target interest points in the corresponding frames of continuous S ₀ or S ₁ are divided and stored in the frame queue, otherwise the consecutive frames corresponding to S ₀ or S ₁ are discarded.

Step (20), in the set of each continuous S ₀ or S ₁ corresponding frames of the frame queue, start to extract continuous key frames from the i-th frame, i>5, so that the number of key frames is the same as that of each section of video classified in the data set The number of frames is the same.

Step (21), the key frame in the step (20) is entered into the video input that the key frame forms successively in the multi-view three-dimensional convolutional neural network trained in the step (15) and employee behavior is classified and identified;

Step (22), comparing the behavior category obtained in step (21) with the predefined standard workflow.

The beneficial effects of the present invention are as follows:

The workflow recognition method based on the three-dimensional convolutional neural network provided by the present invention is mainly composed of the following functional modules: a moving object segmentation module, a behavior recognition module and an action division module.

The moving target segmentation module mainly realizes the segmentation of target interest points from images and video sequences. Since the target motion in the workflow video sequence is relatively large, and the background is basically in a static state, the two frames of images before and after can be subtracted to obtain the inter-frame difference map, and then the motion can be adjusted according to the relationship between the pixel difference and the threshold value target segmentation. The adaptive three-frame difference method adopted is to perform an AND operation on the inter-frame difference maps obtained from the first two frames and the last two frames of the three video frames to obtain a three-frame difference map, and the threshold is set according to the previous The inter-frame difference map is automatically adjusted, so it can effectively avoid the influence of noise;

The behavior recognition module uses 3D convolutional neural network and multi-view learning ability to recognize the behavior of moving targets. To achieve multi-view fusion, we employ a view-pooling layer to fuse these global view information. In multi-view 3D-CNNs, multiple independent 3D-CNNs are used to extract features from image sequences of different views; then, in the view pooling layer, the feature descriptors extracted from different views are fused and learn view correlation. Features; Finally, a fully connected neural network (full connected neural network, FNN) with a softmax classifier is used for final identification;

The action division module defines two states: motion state and relative static state. For the point of interest in each frame, its center coordinates are taken. When the point of interest moves, its center coordinates will also move accordingly. At this time, the difference between the center coordinates of the point of interest in two adjacent frames can be taken to represent the state of the current point of interest. In this way, dynamic and static divisions are realized;

The workflow identification method provided by the present invention can effectively solve two problems that need to be solved in workflow identification in a complex environment. The first problem is that various machines, transport vehicles, auxiliary equipment and other objects in the production workshop are blocked from each other, and different process operations similarity, the impact of frequent light intensity changes in the workshop on workflow recognition, and the second problem is how to automatically divide the production process/action in the image sequence in the video.

Description of drawings

Figure 1 is a schematic diagram of the construction of a multi-view three-dimensional convolutional neural network;

FIG. 2 is a schematic diagram of workflow action division.

detailed description

The present invention will be further described below in conjunction with drawings and embodiments.

First, define the concept and explain the symbols:

Frame Difference Threshold t represents the current frame number, l≥1 represents the recursive order, d _k is the pixel value of the kth pixel in the inter-frame difference map, max{d _k } is the maximum value of the pixel value in the inter-frame difference map, min{d _k } is the minimum value of the pixel value in the inter-frame difference map;

N ₁ and N ₂ respectively represent the satisfaction of with the total number of pixels.

a: Weighted average feature map after weighted average view pooling operation.

t ₁ : The sequence number of the pooled feature map after convolution and pooling operations.

The weight of the pooled feature map corresponding to the serial number t ₁ .

The pooled feature map corresponding to the serial number t ₁ .

Secondly, a workflow recognition method based on a three-dimensional convolutional neural network, the implementation steps are as follows:

(1) Segmentation of moving objects: Video monitoring equipment on the production line is often set up at a higher position, resulting in most areas of the monitoring screen being factory backgrounds that have nothing to do with workflow recognition. If feature vectors are directly extracted from the entire monitoring screen, This will greatly increase the difficulty of feature extraction and the time consumption of calculation. Therefore, the three-frame difference method with adaptive threshold is used to segment the moving target (point of interest) part in the video, thereby reducing the workload of the following steps. specific:

(1.1) Export the multi-view workflow video from the data set, and obtain the video resolution and frame number of each view workflow video;

(1.2) Initialize the inter-frame difference thresholds of the workflow videos of each view; set t=2, and perform steps (1.3) to (1.9) for the workflow videos of each view respectively

(1.3) Read a video frame t and its adjacent two frames t-1 and t+1, and carry out grayscale and median filter processing of these three video frames;

(1.4) Perform inter-frame difference calculations on the first two frames and the last two frames respectively to obtain two inter-frame difference maps;

(1.5) Dynamically update the inter-frame difference threshold according to the two inter-frame difference images obtained in step (1.4), the update method is as follows:

(1.5.1) Set the value of l to 1, and the inter-frame difference threshold of the tth frame d _k is the pixel value of the kth pixel in the inter-frame difference image, max{d _k } is the maximum value of the pixel value in the inter-frame difference image, and min{d _k } is the minimum value of the pixel value in the inter-frame difference image;

(1.5.2) order N ₁ and N ₂ respectively represent the satisfaction of with The total number of pixels;

(1.5.3) If then will Assign value to τ ¹ _t , otherwise, let l=l+1, repeat step (1.5.2);

(1.6) Binarize the current frame (that is, the middle frame) according to the inter-frame difference threshold obtained in (1.5), set the pixel points greater than the inter-frame difference threshold to 1, and set the pixel points smaller than the inter-frame difference threshold to 0;

(1.7) Perform AND operation on the two difference images before and after to obtain three frames of difference images, and use the Blob Extraction method to obtain the center coordinates of the point of interest;

(1.8) Segment the extracted interest points from the original image of the current frame;

(1.9) The value of t is gradually increased by 1, and steps (1.3)-(1.8) are repeated until the value of t is 1 smaller than the value of the last frame of the workflow video. During the repetition, the segmentation size of step (1.8) remains unchanged; The point-of-interest images obtained in step (1.8) in each repetition process are saved as point-of-interest videos in sequence, and the point-of-interest videos are classified according to the classification rules in the data set;

(2) Behavior recognition based on multi-view 3D convolutional neural network: After inspecting the current manufacturing production line, it can be found that in the current manufacturing production line, multiple cameras are often used for synchronous real-time monitoring of the same working scene from different angles. In order to ensure the quality of products and the safety of employees. Taking advantage of this feature, we use the method of multi-view feature extraction and fusion to effectively reduce the impact of the complex environment of the factory on behavior recognition and improve the accuracy of behavior recognition. The specific execution steps are as follows:

(2.1) Select 90% from the point-of-interest videos obtained in (1) as a training set, and the rest as a test set;

(2.2) Construct a multi-view three-dimensional convolutional neural network (see Figure 1). The number of initial training rounds is 5000, and the construction method of the multi-view 3D convolutional neural network is as follows:

The convolution and pooling operation process is (2.2.1) ~ (2.2.4):

(2.2.1) Initialize a four-dimensional convolution kernel with a size of 9*9*9*10 for the first convolutional layer, the activation function is sigmoid, the window size of the first pooling layer is 2, and the step size is 2;

(2.2.2) Initialize a four-dimensional convolution kernel with a size of 9*9*7*30 for the second convolutional layer, the activation function is sigmoid, the window size of the second pooling layer is 2, and the step size is 2;

(2.2.3) Initialize a four-dimensional convolution kernel with a size of 9*8*5*50 for the third convolutional layer, the activation function is sigmoid, the window size of the third pooling layer is 2, and the step size is 2;

(2.2.4) Initialize a four-dimensional convolution kernel with a size of 4*3*3*150 for the fourth convolutional layer, the activation function is sigmoid, the window size of the fourth pooling layer is 2, and the step size is 2;

(2.2.5) Initialize the weight parameters of each feature map in the weighted average view pooling layer is a random value in [0,1], and The weighted average view-pooling layer (WAVP) calculation formula is as follows:

(2.2.6) Initialize a 3000*1500 and 1500*750 convolution kernel for the first two fully connected layers, and set the activation function to Relu; the weighted average feature map after the weighted average view pooling operation is input to the first two layers fully connected layer;

(2.2.7) Initialize a 750*14 convolution kernel for the last fully connected layer and set the Softmax classification function, where 14 is the type of action.

(2.3) Randomly select 20 videos from the training set of workflow videos from each perspective and input them into the multi-view 3D convolutional neural network in (2.2) for feature training, and output the training error;

(2.4) Randomly select 10 videos from the training set of the workflow videos of each view and input them into the multi-view 3D convolutional neural network for verification, and obtain the accuracy rate of classification and recognition of the multi-view 3D convolutional neural network;

(2.5) Repeat (2.3)～(2.4), and the number of training rounds is reduced by 1 every time it is repeated, until the number of training rounds is 0, and a trained multi-view 3D convolutional neural network is obtained;

(2.6) Test the multi-view 3D convolutional neural network in (2.5) using the test set corresponding to the workflow video of each view;

(3) State-based action division method: In the actual environment, the actions of workers often occur continuously. In this case, if the action is to be recognized, the action needs to be segmented first, and then each action can be identified separately. . It is found through observation that a certain displacement (see accompanying drawing 2) will take place in the middle of these behaviors of workers from taking parts, handling parts to placing parts and workers from taking welding tools to welding parts. Therefore, the actions can be divided according to the worker's motion state. The specific execution steps are as follows:

(3.1) For the newly input video, obtain the video resolution and the number of frames, initialize the difference threshold between frames; set t=2;

(3.2) According to (1.3)～(1.7), extract the center coordinates of two adjacent frames of interest points, and calculate the distance between the two center coordinates. If the distance is greater than the threshold T we set, it will be marked as the motion state S ₁ , Otherwise, it is marked as relatively static state S ₀ ;

(3.3) The value of t is gradually increased by 1, repeat (3.2) until the value of t is 1 smaller than the value of the last frame of the newly input video, and the number of continuous S ₀ and S ₁ is counted, when continuous S ₀ or When the number of S ₁ is greater than or equal to N, N>10, use the method (1.8) to segment out the target interest points in the corresponding frame of S ₀ or S ₁ and store them in the frame queue, otherwise discard the corresponding S ₀ or S ₁ frame.

(3.4) In the set of frames corresponding to each continuous S ₀ or S ₁ in the frame queue, extract continuous key frames starting from the i-th frame, i>5, so that it is the same as the number of frames of each segment of video classified in the data set.

(3.5) Input the video composed of key frames in (3.4) into the multi-view three-dimensional convolutional neural network trained in (2.5) to classify and identify employee behavior;

(3.6) Compare the behavior categories obtained in (3.5) with the pre-defined standard workflow.

Claims (1)

1. A workflow recognition method based on a three-dimensional convolutional neural network, characterized in that: the specific steps of the method are: Step (1), export workflow videos containing multiple perspectives from the data set, and obtain the video resolution and frame number of the workflow videos of each perspective; Step (2), initializing the inter-frame difference thresholds of the workflow videos of each perspective; performing steps (3) to (11) for the workflow videos of each perspective; Step (3), setting t=2; Step (4), read three consecutive video frames of t-1, t, and t+1 and carry out grayscale and median filter processing of these three video frames; Step (5), performing inter-frame difference operations on the first two frames and the last two frames respectively to obtain two inter-frame difference images; Step (6), dynamically update the inter-frame difference threshold according to the two inter-frame difference images obtained in step (5); the inter-frame difference threshold dynamic update method is as follows: 6.1 Set l=1, the inter-frame difference threshold of the tth frame d _k is the pixel value of the kth pixel in the inter-frame difference image, max{d _k } is the maximum value of the pixel value in the inter-frame difference image, and min{d _k } is the minimum value of the pixel value in the inter-frame difference image; N ₁ and N ₂ respectively represent the satisfaction of with The total number of pixels; 6.3 If then will Assign value to τ ¹ _t , otherwise, let l=l+1, repeat step 6.2; In step (7), the current frame is binarized according to the inter-frame difference threshold obtained in (6), and the pixels greater than the inter-frame difference threshold are set to 1, and the pixels less than the inter-frame difference threshold are set to 0; Step (8), running and operating the front and rear two frame difference images to obtain three frame difference images, and using the block extraction method to obtain the central coordinates of the point of interest; Step (9), segmenting the extracted point of interest from the original image of the current frame; Step (10), the value of t is gradually increased by 1, and steps (4) to (9) are repeated until the value of t is 1 smaller than the value of the last frame of the workflow video, and the segmentation size of step (9) remains unchanged during the repetition process ; The point-of-interest image obtained in step (9) in each repetition process is preserved as the point-of-interest video in sequence, and the point-of-interest video is classified according to the classification rules in the data set; Step (11), randomly select 90% from the point-of-interest video obtained in step (10) as a training set, and the rest as a test set; Step (12), construct a multi-view three-dimensional convolutional neural network, and initialize the number of training rounds to be 5000; the construction method of the multi-view three-dimensional convolutional neural network is as follows: 12.1. The convolution and pooling operations are as follows: ① Initialize a four-dimensional convolution kernel with a size of 9*9*9*10 for the first convolutional layer, the activation function is sigmoid, the window size of the first pooling layer is 2, and the step size is 2; ②Initialize a four-dimensional convolution kernel with a size of 9*9*7*30 for the second convolutional layer, the activation function is sigmoid, the window size of the second pooling layer is 2, and the step size is 2; ③Initialize a four-dimensional convolution kernel with a size of 9*8*5*50 for the third convolutional layer, the activation function is sigmoid, the window size of the third pooling layer is 2, and the step size is 2; ④ Initialize a four-dimensional convolution kernel with a size of 4*3*3*150 for the fourth convolutional layer, the activation function is sigmoid, the window size of the fourth pooling layer is 2, and the step size is 2; 12.2. Initialize the weight parameters of each feature map in the weighted average view pooling layer is a random value in [0,1], and The weighted average view pooling operation in the weighted average view pooling layer is as follows: <mrow> <mi>a</mi> <mo>=</mo> <mfrac> <mrow> <msub> <mi>&amp;Sigma;</mi> <msub> <mi>t</mi> <mn>1</mn> </msub> </msub> <msup> <mi>exp</mi> <msup> <mi>&amp;alpha;</mi> <msub> <mi>t</mi> <mn>1</mn> </msub> </msup> </msup> <mo>&amp;CenterDot;</mo> <msup> <mi>z</mi> <msub> <mi>t</mi> <mn>1</mn> </msub> </msup> </mrow> <mrow> <msub> <mi>&amp;Sigma;</mi> <msub> <mi>t</mi> <mn>1</mn> </msub> </msub> <msup> <mi>exp</mi> <msup> <mi>&amp;alpha;</mi> <msub> <mi>t</mi> <mn>1</mn> </msub> </msup> </msup> </mrow> </mfrac> </mrow> In the formula, a is the weighted average feature map after the weighted average view pooling operation, t ₁ is the serial number of the pooled feature map after the convolution and pooling operation, is the weight of the pooled feature map corresponding to the serial number t ₁ , exp represents the exponential function with e as the base, is the pooled feature map corresponding to the serial number t ₁ ; 12.3. Initialize a convolution kernel of 3000*1500 and 1500*750 for the first two fully connected layers, and set the activation function to Relu; the weighted average feature map after the weighted average view pooling operation is input to the first two fully connected layers ; 12.4. Initialize a 750*14 convolution kernel for the last fully connected layer and set the Softmax classification function; In step (13), 20 videos are randomly selected from the training set corresponding to the workflow videos of each perspective and input into the multi-view 3D convolutional neural network in (12) for feature training, and the training error is output; Step (14), randomly select 10 videos from the training set corresponding to the workflow videos of each perspective and input them into the multi-view three-dimensional convolutional neural network for verification, and obtain the accuracy rate of classification and recognition of the multi-view three-dimensional convolutional neural network; Step (15), repeating steps (13) to (14), the number of training rounds is reduced by 1 each time it is repeated, until the number of training rounds is 0, and a trained multi-view three-dimensional convolutional neural network is obtained; Step (16), testing the multi-view three-dimensional convolutional neural network in step (15) using the test set corresponding to the workflow video of each perspective; Step (17), for the newly input workflow video, obtain the video resolution and the number of frames, and initialize the difference threshold between frames; set t=2; Step (18), extract the center coordinates of two adjacent frames of interest points according to steps (4) to (8), and calculate the distance between the two center coordinates, if the distance is greater than the set threshold T, mark it as a motion state S ₁ , otherwise, it is marked as relative static state S ₀ ; Step (19), the value of t is gradually increased by 1, and step (18) is repeated until the value of t is smaller than the value of the last frame of the newly input workflow video by 1, and the number of continuous S ₀ and S ₁ is counted. When the number of S ₀ or S ₁ is greater than or equal to N, the target interest points in the corresponding frames of continuous S ₀ or S ₁ are divided and stored in the frame queue, otherwise the corresponding frames of continuous S ₀ or S ₁ are discarded; Step (20), in the collection of each continuous S ₀ or S ₁ corresponding frames of the frame queue, start to extract continuous key frames from the i-th frame, i>5, so that the number of key frames is the same as that of each section of video classified in the data set same number of frames; Step (21), the key frame in the step (20) is entered into the video input that the key frame forms successively in the multi-view three-dimensional convolutional neural network trained in the step (15) and employee behavior is classified and identified; Step (22), comparing the behavior category obtained in step (21) with the predefined standard workflow.