CN101739122A - Gesture Recognition and Tracking Method - Google Patents

Abstract

A method for gesture recognition and tracking uses an image sensor to capture a gesture image, and then processes the image to recognize and track the image, and execute the action corresponding to the gesture image. First, the method pre-processes the image, then performs image motion detection, and then analyzes the characteristics of the image to determine the gesture state of the image. If the gesture image is a moving gesture, a center coordinate of the moving gesture is detected and tracked, and the center coordinate of the moving gesture is output; if the gesture image is a command gesture, the action instruction corresponding to the command gesture is output. To this end, the purpose of natural gesture recognition and tracking is achieved using the digital signal processor as a hardware platform.

Description

Gesture Recognition and Tracking Method

technical field

The invention relates to a gesture recognition and tracking method, in particular to a method for using a digital signal processor to recognize and track the gesture.

Background technique

With the advancement of computer technology, the improvement of human-computer interaction has always been the focus of many research institutes. From the early days of keyboards, mice and joysticks, they are all designed to allow users to operate computers more conveniently. In many virtual reality (virtual reality) and multimedia system (multimedia system) applications, for 3D object manipulation, 3D virtual product display systems, computer graphics systems, and action or sports video games... and other applications, must Input device with 3D and high degree of freedom. However, the aforementioned input devices such as keyboards, mice, and joysticks cannot conveniently and properly provide natural and direct interaction effects between the user and the system.

Since the application of human computer interface (human computer interface) is becoming more and more popular, including gesture recognition, voice recognition, or body language recognition, etc., it has been widely studied and applied in daily life. The most natural and direct. Therefore, applying gesture recognition to machine vision and virtual reality has become a new development trend.

In the practical application of gesture recognition and gesture tracking as a computer input interface, the use of a glove-based method can provide accurate and rapid sensing and recognition effects. The so-called glove-based method means that the user needs to wear a data glove, and the data glove is equipped with a contact sensor, which can accurately capture the curvature of the user's finger or hand movement, and can move the hand activity is transmitted to the computer as an electronic signal. By analyzing these electronic signals, the system can quickly identify the action state of the gesture. However, the unit price of such data gloves equipped with contact sensors is quite expensive, and the size specifications of data gloves are not diverse, causing users to have more choices in terms of wearing fit, and wearing heavy data gloves. Gloves also tend to make hands feel tired and restrict the user's activities, resulting in inconvenience in use.

Therefore, how to design a gesture recognition and tracking method that can reduce development costs and simplify operating procedures, shorten the distance between the user and the machine, and make the man-machine interface more efficient, more humane and more Moving in the direction of diversification is a major issue to be overcome and resolved in this case.

Contents of the invention

In view of this, the present invention provides a gesture recognition and tracking method, using an image sensor to capture a gesture image, and then processing the image by a digital signal processor to identify and track the gesture image, and execute The action command corresponding to the gesture image. Therefore, the purpose of using the digital signal processor as a hardware platform to recognize and track natural gestures is achieved.

In order to solve the above problems, the present invention provides a gesture recognition and tracking method. The steps of the method are as follows: firstly, performing pre-processing on the gesture image. Then, a maximum motion block of the gesture image is detected and defined as a gesture block. Then, the feature of the gesture block is analyzed, and then it is determined that the gesture block is a movement confirmation gesture, a command gesture or other undefined gestures. Finally, if the gesture block is the movement confirmation gesture, then transformed into a movement gesture, and the movement gesture does not stop for more than an action time during the movement, then detect and track a center coordinate of the movement gesture, and output the The center coordinates of the move gesture.

The beneficial effect of the present invention is that the purpose of recognizing and tracking natural gestures can be achieved by using the digital signal processor as the hardware platform.

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, but not as a limitation of the present invention.

Description of drawings

FIG. 1 is a flowchart of a gesture recognition and tracking method of the present invention;

Fig. 2 is a schematic diagram of dynamic image differential calculation of the present invention;

Fig. 3 is the schematic diagram of horizontal and vertical projection amount calculation of the present invention;

4A to 4C are schematic diagrams of the block labeling process of the present invention;

5A to 5B are schematic diagrams of the process of continuously tracking the coordinates of the center of the moving gesture in the present invention; and

FIG. 6 is a block diagram of a gesture recognition and tracking device of the present invention.

Among them, reference signs

10 image sensor

20 digital signal processor

30 first memory

40 second memory

50 video output modules

60 data input/output modules

Ps template

S102-S502 steps

Detailed ways

Relevant technical content and detailed description of the present invention, cooperate drawing description as follows:

Please refer to FIG. 1 , which is a flowchart of a gesture recognition and tracking method of the present invention. In the method, an image sensor is used to capture a gesture image, and then a digital signal processor is used to process the gesture image. The steps of the method are described in detail as follows, and an application in which a gesture action replaces a mouse action is taken as an example.

First, the digital signal processor performs pre-processing on the gesture image (S102). Because unprocessed images usually contain some noise, which increases the probability of identification errors, and too much useless information will also reduce the efficiency of the overall execution. Therefore, the captured images will be pre-processed before analysis . The image pre-processing step (S102) is as follows: first adjust the size of the gesture image to fit the calculation range; then perform color conversion on the gesture image, and reduce the image from a full-color image (24bit RGB) to a grayscale image (8bit gray level); finally, the point-like noise of the gesture image is filtered out by an image low pass filter to facilitate subsequent actual calculations. In this way, image pre-processing not only increases the accuracy of recognition, but also saves data storage space and improves transmission speed.

Then detect a maximum motion block of the gesture image and define it as a gesture block ( S104 ). The image motion detection step (S104) is as follows in sequence: First, use dynamic image difference to generate binary images, and use logic operations on these binary images to calculate all moving parts in the gesture image; and the number of horizontal bright spots, and select the logic of the maximum projection area of the vertical axis and the horizontal axis to find the maximum moving area of the gesture image; The detailed fragmented images of the logical area are filled; finally, the label number is used to exclude the non-gesture motion area, and the maximum connected area is reserved and calculated to detect the maximum motion block.

Please refer to FIG. 2 for a schematic diagram of dynamic image difference calculation in the present invention. As shown in the figure, the real moving object is calculated by using three consecutive frames of gesture images. Wherein, the current grayscale image frame is marked as M2, the previous grayscale image frame is marked as M1, and the previous two grayscale image frames are marked as M0. In addition, a threshold value is set to serve as a basis for converting the grayscale images into binary images. First subtract the previous grayscale image M1 from the current frame grayscale image M2 to obtain a new grayscale image, and then compare the pixels of the new grayscale image with the gate threshold: if the grayscale image’s If the pixel is greater than or equal to the gate threshold, it is set as a bright spot; if the pixel of the grayscale image is smaller than the gate threshold, it is set as a dark spot, and a new binary image M3 is obtained. Similarly, the previous two frames of gray-scale image M0 are subtracted from the previous frame of gray-scale image M1 to obtain another new gray-scale image, and then compared with the gate threshold to obtain another new binary image M4. Finally, logical (AND) operation is performed on the two binary images M3 and M4 to obtain the binary image M5 of the last moving part.

Please refer to FIG. 3 which is a schematic diagram of calculation of horizontal and vertical projection amounts in the present invention. Count the number of vertical bright spots and the number of horizontal bright spots on the obtained binary image M5 of the last moving part to find out the maximum moving area. As shown in the figure, there are two large movement blocks, marked as X and Y respectively. After calculation, two large areas A and B are obtained by the horizontal projection amount; two large areas C and D are obtained by the vertical projection amount. area. Then, the logical area of the maximum projection amount area B on the horizontal axis and the maximum projection amount area C on the vertical axis can be obtained to obtain the maximum movement amount block X.

Please refer to FIG. 4A to FIG. 4C which are schematic diagrams of the block labeling process of the present invention. After the detailed fragmented images in the logical area of the maximum projection amount on the vertical and horizontal axes are filled, the bright and dark points of the binary image are represented by binary values (marked as 0 and 1 in FIG. 4A ). Then, renumber the connected regions and calculate the area, and finally only keep the largest part (the region marked as 2 in FIG. 4B and FIG. 4C ), and exclude the non-gesture movement region.

Then analyze the feature of the gesture block, and then determine that the gesture block is a movement confirmation gesture, a command gesture or other undefined gestures (S106). The feature comparison of the gesture block is to use the relative extreme point occurrence position and the extreme value difference generated by each gesture to compare with the gesture image data in the database one by one, and store the comparison result in the memory buffer middle. For example, when the operator spreads five fingers, because it is defined that the relative maximum value appears at the fingertips, and the relative minimum value appears at the junction of the webs of the two fingers and the left and right sides of the palm. Therefore, the feature of the gesture block has five maximum values and six minimum values. Therefore, when the gesture block is compared with the gesture image data in the database, it is recognized that the operator's gesture is a state of extending five fingers.

See Figure 1 again. Then if the gesture block is the movement confirmation gesture (S108), it is determined whether to continue to transform into a movement gesture (S200). If the movement gesture is not continued, the step (S102) is re-executed. If it is determined that the operator's gesture is changed from the movement confirmation gesture to the movement gesture, a cursor can be controlled to move. Wherein, the movement confirmation gesture can be defined as a V shape formed by the index finger and the middle finger. When the operator stretches out the index finger and the middle finger to form the V shape, after comparing the gesture block with the gesture image data in the database, Then it is recognized that the operator's gesture is a state of extending the index finger and the middle finger, and the gesture is confirmed for the movement. When the moving gesture is in a moving state, an action of moving the cursor is generated. Then it is judged whether the moving gesture continues to stop moving for more than an action time ( S300 ). If the operator stops the gesture movement for more than the action time, step (S102) is re-executed. Wherein, the action time can be set to different lengths of time according to the operator's use mode or needs, for example, the action time can be set to 1 second. If the operator stops the movement of the gesture within the action time, it is determined whether a center coordinate of the movement gesture is detected (S400). If the center coordinate of the movement gesture is not detected, re-detect the center coordinate of the movement gesture (S404), and then perform step (S400) to re-determine whether the center coordinate of the movement gesture is detected. Since the movement gesture is defined as a fist shape formed by clenching a fist with five fingers, when the center coordinates are started to be tracked, the circular Hough transfer is used to statistically find the most points with the same center of circle, that is, Determine where the center coordinates are located.

If the center coordinate of the moving gesture is detected, a fast integration table (sum ofaccumulator table, SAT) method (S402) is used to determine whether the center coordinate of the moving gesture is tracked (S500). If the center coordinate of the moving gesture is not tracked, step (S404) is executed again to re-detect the center coordinate of the moving gesture. If the center coordinate of the movement gesture is tracked, output the center coordinate of the movement gesture (S502), and then re-execute the step (S102).

Please refer to FIG. 5A to FIG. 5B for schematic diagrams of the process of continuously tracking the center coordinates of the moving gesture in the present invention. When the center coordinates of the moving gesture are detected, take 20 pixels from the center coordinates up, down, left, and right to generate a template Ps with a size of 40 pixels*40 pixels and calculate the sum of the grayscale values of all pixels, and then Then use the fast integral table method to search one by one in the area of plus or minus 60 pixels of the center coordinates for the position where the difference of the total value of the comparison is the smallest or equal, which is the new center position of the movement gesture. At this time, the new coordinates are generated and stored in the memory buffer and move the search area to the new center coordinates. In this way, the square template Ps is compared one by one in the search area from the upper left to the lower right, so as to keep tracking the center coordinates. Finally, if the moving gesture stops and lasts longer than the action time, the tracking is ended and the step ( S102 ) is re-executed.

In addition, in step S106, if the gesture block is detected as the command gesture (S110), an action instruction corresponding to the command gesture is output (S112), and then step (S102) is re-executed. For example, the command gesture can be defined as a 1 shape formed by the index finger, and the action instruction corresponding to the command gesture is a click action. When the operator stretches out the index finger to form a 1 shape, after comparing the gesture block with the gesture image data in the database, it is recognized that the operator’s gesture is in the state of extending the index finger. Therefore, at the coordinates where the cursor is located Perform a click action. The command gesture can be defined as other gesture shapes according to the operator's operating habits, or other valid command gestures can be defined to execute respective corresponding action commands.

In addition, in step S106, if the gesture block is detected as the undefined gesture (S114), that is, the command gesture is not the movement confirmation gesture (a V shape formed by the index finger and the middle finger), the movement gesture (five-finger A fist shape formed by clenching a fist) or any gesture of the command gesture (a 1-shaped shape formed by the index finger), if it is an invalid undefined gesture, then re-execute the step (S102).

Please refer to FIG. 6 which is a block diagram of a gesture recognition and tracking device of the present invention. The device includes an image sensor 10 , a digital signal processor 20 , a first memory 30 , a second memory 40 and a video output module 50 .

The image sensor 10 is used to capture a gesture image. The digital signal processor 20 is electrically connected to the image sensor 10 for providing an algorithm to process the gesture image. The first memory 30 is electrically connected to the digital signal processor 20 for storing the algorithm of the digital signal processor 20 and providing storage of a large amount of calculation data. Wherein, the first memory 30 can be a flash memory (flash memory). The second memory 40 is electrically connected to the digital signal processor 20 to provide a memory buffer required by the digital signal processor 20 for operation. Wherein, the second memory 40 can be a random access memory (random access memory). The video output module 50 is electrically connected to the digital signal processor 20 for outputting an image calculation result obtained by the digital signal processor 20 . Wherein, the image calculation result can be output to an analog display device (not shown), such as a TV or a monitor; or a digital display device (not shown), such as a liquid crystal display. The digital signal processor 20 can be further electrically connected to a data input/output module 60, so as to not only transmit the image calculation result to other devices (not shown), such as independent operating devices such as computers or video game hosts, through different output interfaces, At the same time, it also accepts external control commands to adjust the operation of the digital signal processor 20 .

In summary, the present invention has the following advantages:

1. Use the digital signal processor as a hardware platform to recognize and track natural gestures without wearing additional gloves or special icons, colors or light-emitting devices, which can greatly reduce development costs and simplify operating procedures.

2. The hardware platform of the digital signal processor can be connected to other external independent devices, which improves the convenience of portability and flexible expansion of applications.

Certainly, the present invention also can have other multiple embodiments, without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and deformations according to the present invention, but these corresponding Changes and deformations should belong to the scope of protection of the appended claims of the present invention.

Claims (13)

1. the method for gesture identification and tracking utilizes an image sensor to capture a gesture image, by this gesture image is handled, it is characterized in that again the step of this method comprises:

(a) this gesture image is carried out pre-process;

(b) detect a largest motion block of this gesture image and be defined as a gesture block;

(c) analyzing the feature of this gesture block, is to move to confirm a gesture or an order gesture to judge this gesture block;

(d) if this gesture block, judge then whether this gesture block continues for should move confirming gesture and be transformed to one and move gesture;

(e) if this gesture block continues and is transformed to this and moves gesture, and this mobile gesture does not stop then to detect and follow the trail of the centre coordinate that this moves gesture above an actuation time in moving process; And

(f) export this centre coordinate that this moves gesture, and re-execute step (a).

2. the method for gesture identification according to claim 1 and tracking is characterized in that, this step (a) comprises:

(a1) adjust this gesture image size for being fit to the calculation scope;

(a2) this gesture image is carried out color conversion; And

(a3) the point-like noise of this gesture image of filtering.

3. the method for gesture identification according to claim 1 and tracking is characterized in that, this step (b) comprises:

(b1) utilize the dynamic image difference, calculate all movable parts in this gesture image;

(b2) utilize the vertical and horizontal bright spot quantity of adding up this gesture image, find out the maximum moving area of this gesture image;

(b3) utilize expansion technique, calculate the broken image of thin portion and fill up; And

(b4) use tag number, calculate largest connected zone to detect this largest motion block.

4. the method for gesture identification according to claim 1 and tracking is characterized in that, this step (e) comprises:

(e1) utilize circular Hough conversion, detect this centre coordinate that this moves gesture; And

(e2) utilize the quick point table, follow the trail of this centre coordinate that this moves gesture.

5. the method for gesture identification according to claim 1 and tracking is characterized in that, in step (d), if this gesture block was confirmed gesture for moving, but the conversion that continues is non-for this mobile gesture, then re-executes step (a).

6. the method for gesture identification according to claim 1 and tracking is characterized in that, in step (d), if this gesture block is then exported the pairing action command of this order gesture, and re-executed step (a) for this order gesture.

7. the method for gesture identification according to claim 1 and tracking is characterized in that, in step (d), if this gesture block is a undefined gesture, then re-executes step (a).

8. the method for gesture identification according to claim 1 and tracking is characterized in that, in step (e), if this mobile gesture stops to surpass this actuation time, then re-executes step (a) in moving process.

9. the method for gesture identification according to claim 1 and tracking is characterized in that, in step (e), if can't detect side or track this centre coordinate that this moves gesture, this moves this centre coordinate of gesture then to detect tracking again.

10. the method for gesture identification according to claim 1 and tracking is characterized in that be set at 1 second this actuation time.

11. the method for gesture identification according to claim 1 and tracking is characterized in that, this moves confirms that gesture is defined as a V-shape of forefinger and middle finger formation.

12. the method for gesture identification according to claim 1 and tracking is characterized in that, this moves gesture and is defined as the fist shape that the five fingers are clenched fist and formed.

13. the method for gesture identification according to claim 1 and tracking is characterized in that, this order gesture is defined as one 1 word shapes that forefinger forms.

Images