CN111428729A - Target detection method and device - Google Patents

Abstract

The invention discloses a target detection method and device, and relates to the technical field of computers. One embodiment of the method comprises: collecting a 2D image and a 3D depth image of a detection target; taking the 3D depth image as masking data, and processing a first feature map generated by the 2D image to obtain a second feature map; and inputting the second characteristic diagram into a target detection network to obtain the category information and the position information corresponding to the detection target. The embodiment can improve the identification degree of the detection target and each plane thereof, improve the detection accuracy, and can save the detection cost without increasing extra calculation burden.

Description

A target detection method and device

technical field

The present invention relates to the field of computer technology, and in particular, to a target detection method and device.

Background technique

With the advancement of technology and the development of mechanical automation, more and more warehouses have begun to use robots to grab goods for sorting to save manpower. In the current situation that the control system of the manipulator is relatively mature, the detection and positioning of the grasped target has always been a difficult point.

Existing object detection schemes include:

SIFT (Scale-invariant feature transform) is a feature plus template matching scheme, which needs to establish a template for the detection target in advance. Taking the detection of warehouse goods as an example, there are thousands of goods in the warehouse, and the cost of establishing a template for each type of goods is very high. At the same time, it is difficult to extract SIFT features for some goods without texture, which makes it difficult to identify;

The edge detection scheme is very sensitive to changes in illumination and texture interference on the detection target (such as a cargo packaging box), and the recognition degree of each plane of the detection target is low.

In the process of realizing the present invention, the inventor found that there are at least the following problems in the prior art:

The existing solution has low recognition degree of the detection target and its planes, low detection accuracy, and high detection cost.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present invention provide a target detection method and device, which can improve the recognition of the detection target and its planes, improve the detection accuracy, do not increase additional computational burden and save detection costs.

To achieve the above object, according to an aspect of the embodiments of the present invention, a target detection method is provided.

A target detection method, comprising: collecting a 2D (two-dimensional) image and a 3D (three-dimensional) depth image of a detection target; using the 3D depth image as masking data, and processing a first feature map generated from the 2D image , obtain a second feature map; input the second feature map into a target detection network to obtain category information and location information corresponding to the detection target.

Optionally, the step of using the 3D depth image as masking data to process the first feature map generated by the 2D image to obtain a second feature map includes: subjecting the 2D image to N-level convolution and Pooling processing to obtain N first feature maps; performing N-level pooling processing on the 3D depth images to obtain N pooled 3D depth images, the pooled 3D depth images and the first feature maps One-to-one correspondence, the N is a positive integer; for each of the first feature maps, the corresponding pooled 3D depth image is used as masking data, and preset processing is performed with the features in the first feature map to obtain the second feature map.

Optionally, the preset processing includes one of dot product, addition and combination.

According to another aspect of the embodiments of the present invention, a target detection apparatus is provided.

A target detection device, comprising: an image acquisition module for acquiring a 2D image and a 3D depth image of a detection target; a feature map processing module for using the 3D depth image as masking data, The first feature map is processed to obtain a second feature map; the target detection module is used for inputting the second feature map into a target detection network to obtain category information and position information corresponding to the detection target.

Optionally, the feature map processing module is further configured to: perform N-level convolution and pooling processing on the 2D image to obtain N first feature maps; perform N-level pooling processing on the 3D depth image, Obtain N pooled 3D depth images, the pooled 3D depth images are in one-to-one correspondence with the first feature map, and N is a positive integer; for each first feature map, the corresponding pooled The transformed 3D depth image is used as masking data, and pre-processed with the features in the first feature map to obtain the second feature map.

Optionally, the preset processing includes one of dot product, addition and combination.

According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

An electronic device comprising: one or more processors; a memory for storing one or more programs that, when executed by the one or more processors, cause the one or more programs A plurality of processors implement the target detection method provided by the present invention.

According to yet another aspect of the embodiments of the present invention, a computer-readable medium is provided.

A computer-readable medium on which a computer program is stored, and when the program is executed by a processor, implements the target detection method provided by the present invention.

One embodiment of the above invention has the following advantages or beneficial effects: collecting a 2D image and a 3D depth image of the detection target; using the 3D depth image as masking data, and processing the first feature map generated from the 2D image to obtain the first feature map. Two feature maps; input the second feature map into the target detection network to obtain category information and location information corresponding to the detected target. The identification of the detection target and its respective planes can be improved, the detection accuracy can be improved, the additional calculation burden is not increased, and the detection cost can be saved.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

1 is a schematic diagram of main steps of a target detection method according to a first embodiment of the present invention;

2 is a schematic structural diagram of a residual block according to an embodiment of the present invention;

3 is a schematic structural diagram of a target detection model according to a second embodiment of the present invention;

4 is a schematic structural diagram of a target detection model according to a third embodiment of the present invention;

5 is a schematic diagram of a product detection process according to a fourth embodiment of the present invention;

6 is a schematic diagram of main modules of a target detection device according to a fifth embodiment of the present invention;

FIG. 7 is an exemplary system architecture diagram to which an embodiment of the present invention may be applied;

FIG. 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

As will be appreciated by those skilled in the art, embodiments of the present invention may be implemented as a system, apparatus, device, method or computer program product. Accordingly, the present disclosure may be embodied in entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

FIG. 1 is a schematic diagram of main steps of a target detection method according to an embodiment of the present invention.

As shown in FIG. 1 , a target detection method according to an embodiment of the present invention mainly includes the following steps S101 to S103.

Step S101: Collect a 2D image and a 3D depth image of the detection target.

The detection target in the embodiment of the present invention may be a target to be detected and identified in various application scenarios, for example, in a warehouse picking scenario, the detection target is a commodity grasped by a robotic arm.

The 2D image of the detection target is the 2D color image (RGB image) of the detection target, and the 2D color camera and the 3D depth camera (the two cameras may be the same integrated camera) can be used to collect the 2D image and the 3D depth image respectively.

After step S101, the collected 2D image and 3D depth image of the detection target can be aligned, and the missing value part on the 3D depth image can be blacked out. The black area on the 3D depth image will not affect the detection target. identification of the area.

Step S102: Using the 3D depth image of the detection target as mask data, process the first feature map generated from the 2D image of the detection target to obtain a second feature map.

Step S102 may specifically include:

Convolve and pool the 2D image of the detection target through N-level residual blocks and pooling layers to obtain N first feature maps;

Perform N-level pooling processing on the 3D depth image of the detection target to obtain N pooled 3D depth images, and the pooled 3D depth images correspond one-to-one with the first feature map;

For each first feature map, the corresponding pooled 3D depth image is used as masking data, and a dot product is performed with the features in the first feature map to obtain a second feature map.

where N is a positive integer.

The structure of the residual block (resblock) can be shown in Figure 2. As shown in Figure 2, this structure includes a convolution layer with 64 convolution kernels and a convolution kernel size of 1*1, a convolution layer with 64 convolution kernels A convolutional layer with kernel size 1*3 and a convolutional layer with 256 kernels and kernel size 1*1. relu represents the activation function.

As an alternative embodiment, the above-mentioned residual block may be replaced with a block (Block) structure.

As an alternative embodiment, the above-described dot multiplication process may be replaced by an addition process, or a concatenate process.

Step S103: Input the second feature map into the target detection network to obtain category information and position information corresponding to the detection target.

The target detection network can be a convolutional network composed of convolutional layers, or it can use the detection layer or detection module of the existing target detection network, such as the last layer detection layer of SSD (single-shot multi-box detector), or faster rcnn (Faster Regional Convolutional Neural Network) followed by RPN (Regional Candidate Network) detection module, etc.

FIG. 3 is a schematic structural diagram of a target detection model according to a second embodiment of the present invention. In the second embodiment of the present invention, after the 2D image and the 3D depth image of the detection target are collected, the collected 2D image and the 3D depth image are input into the target detection model shown in FIG. 3 . The target detection model includes: a block, a first pooling layer, a second pooling layer, and a detection layer. A block (Block) includes multiple convolutional layers. The block (Block) in this embodiment of the present invention may adopt a resblock (residual block) structure. For the specific structure of the residual block, refer to the introduction in FIG. 2 , which will not be repeated here. The block and the first pooling layer (the first pooling layer is connected after the corresponding block) is used to perform multi-level convolution and pooling on 2D images, and the multi-level second pooling layer is used for 3D depth images. pooling. The levels of the block and the pooling layer in FIG. 3 are only four layers as an example, and the target detection model in the embodiment of the present invention may not be limited to the four-level block and the pooling layer.

The first pooling layer and the second pooling layer at the same level can be implemented by the same pooling layer, that is, the same pooling layer can perform pooling processing on both 2D images and 3D depth images.

The four-level block and the first pooling layer of Figure 3 output four first feature maps (one first feature map per stage), and the second pooling layer outputs four pooled 3D depth images (one pool per stage) 3D depth image of the The first feature maps generated at each stage correspond to the pooled 3D depth images generated at each stage. For each first feature map, take the corresponding pooled 3D depth image as masking data, and perform dot product processing with the features in the first feature map to obtain a second feature map. For example, in Figure 3, use the first feature map. The pooled 3D depth map generated by the second pooling layer 1 is used as masking data, and is subjected to dot product processing with the features of the block + the first feature map generated by the first pooling layer 1 to obtain a second feature map. In the same way, four second feature maps are finally obtained, and the four second feature maps are input to the detection layer for target detection processing to generate detection results, that is, the category information and position information (detection frame information) corresponding to the detection target.

代表点乘，这个结构里，把3D深度图像当成遮罩(mask，即遮掩数据)，对2D彩色图像提出来的特征进行点乘，从而对不同平面的像素值进行了不同的加权，可以极大的提高检测目标及其各平面的辨识度。并且，本发明实施例对3D深度图像仅做了池化(pool)操作，为了和特征(feature)的尺度保持一致，基本没有额外的计算负担，节省检测成本。In Figure 3

Represents point multiplication. In this structure, the 3D depth image is regarded as a mask (mask, that is, masking data), and the features proposed by the 2D color image are dot multiplied, so that the pixel values of different planes are weighted differently, which can be extremely Greatly improve the recognition of the detection target and its planes. In addition, the embodiment of the present invention only performs a pooling operation on the 3D depth image, in order to keep the scale consistent with the feature (feature), there is basically no additional computational burden and the detection cost is saved.

The detection layer in the embodiment of the present invention may use a convolutional network composed of convolutional layers, or may use a detection layer or detection module of an existing target detection network. The detection layer convolves the second feature map respectively, and extracts the category information and detection frame information at the same time.

表示加法处理)，以得到第二特征图。同样地，本实施例的目标检测模型可以不限于四级块和池化层。其他具体实施细节可参加第二实施例的介绍。FIG. 4 shows the structure of the target detection model of the third embodiment of the present invention. The structure of this embodiment is similar to that of the target detection model of the second embodiment, and the difference is that for each first feature map, the corresponding pooled 3D depth image is used as masking data, which is the same as the first feature map. The features of are additively processed (in the figure

represents the addition process) to obtain the second feature map. Likewise, the target detection model of this embodiment may not be limited to the four-level block and pooling layer. For other specific implementation details, please refer to the introduction of the second embodiment.

The target detection model in the embodiment of the present invention is a trained model. In the model training stage, 2D color camera and 3D depth camera can be used to collect the real data (2D image and 3D depth image) of the detection target as training data, and the training data can be manually labeled. ) position and category, then train the parameters of each layer of the model, and finally get the trained target detection model. The model training method can adopt various training methods for target detection models, such as stochastic gradient descent and backpropagation training methods. The embodiment of the present invention trains the target detection model based on the 3D depth image and the 2D color image, which overcomes the defect that the existing target detection model extracts limited information, and the target detection model of the embodiment of the present invention regards the depth as a kind of masking data ( mask), the color image is modified, and then detected, which improves the detection accuracy. Applying the target detection model of the embodiment of the present invention to deployment in the warehouse picking link can greatly improve the detection rate of goods.

The fourth embodiment of the present invention takes commodity detection as an example. As shown in FIG. 5 , the commodity detection process of the fourth embodiment of the present invention includes steps S501 to S504 .

Step S501: Collect 2D images and 3D depth images of the commodity, and obtain training samples through manual annotation.

Step S502: Build a product detection model, which includes a residual block, a pooling layer, and a detection layer.

Step S503: Use the training samples to train the above-mentioned commodity detection model based on the depth image and the color image.

Step S504: Input the 2D image and 3D depth image of the commodity to be detected into the trained commodity detection model to obtain the category and detection frame information of the commodity to be detected.

In this embodiment, the target detection step is described by taking a commodity as an example. For the detailed implementation of each step and the structure of the commodity detection model, reference may be made to the introduction of the other embodiments above.

FIG. 6 is a schematic diagram of main modules of a target detection apparatus according to a fifth embodiment of the present invention.

The target detection device 600 according to the fifth embodiment of the present invention mainly includes: an image acquisition module 601 , a feature map processing module 602 , and a target detection module 603 .

The image acquisition module 601 is used to acquire the 2D image and the 3D depth image of the detection target.

The feature map processing module 602 is configured to process the first feature map generated from the 2D image by using the 3D depth image as mask data to obtain a second feature map.

The feature map processing module 602 is specifically used for:

Convolve and pool the 2D image of the detection target through N-level blocks and pooling layers to obtain N first feature maps;

Perform N-level pooling processing on the 3D depth image of the detection target to obtain N pooled 3D depth images, and the pooled 3D depth images are in one-to-one correspondence with the first feature map, and N is a positive integer;

For each first feature map, the corresponding pooled 3D depth image is used as masking data, and preset processing is performed with the features in the first feature map to obtain a second feature map.

As an alternative embodiment, the block may adopt the structure of a residual block.

Preset processing includes one of dot product, addition, and merge.

The target detection module 603 is configured to input the second feature map into the target detection network to obtain category information and position information corresponding to the detected target.

The target detection network can be a convolutional network composed of convolutional layers, or it can use the detection layer or detection module of the existing target detection network, such as the last layer detection layer of SSD (single-shot multi-box detector), or faster rcnn (Faster Regional Convolutional Neural Network) followed by RPN (Regional Candidate Network) detection module, etc.

In addition, the specific implementation content of the target detection device in the embodiment of the present invention has been described in detail in the above-mentioned target detection method, so the repeated content is not described here.

FIG. 7 shows an exemplary system architecture 700 to which a target detection method or a target detection apparatus according to an embodiment of the present invention may be applied.

As shown in FIG. 7 , the system architecture 700 may include terminal devices 701 , 702 , and 703 , a network 704 and a server 705 . The network 704 is the medium used to provide the communication link between the terminal devices 701 , 702 , 703 and the server 705 . Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 701 , 702 and 703 , such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

The terminal devices 701, 702, 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 705 may be a server that provides various services, such as a background management server that provides support for shopping websites browsed by the terminal devices 701 , 702 , and 703 (just an example). The background management server can analyze and process the received product information query request and other data, and feed back the processing result (for example, product information—just an example) to the terminal device.

It should be noted that the target detection method provided by the embodiment of the present invention is generally executed by the server 705 , and accordingly, the target detection apparatus is generally set in the server 705 .

It should be understood that the numbers of terminal devices, networks and servers in FIG. 7 are only illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Referring next to FIG. 8 , it shows a schematic structural diagram of a computer system 800 suitable for implementing the server of the embodiment of the present application. The server shown in FIG. 8 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

As shown in FIG. 8, a computer system 800 includes a central processing unit (CPU) 801, which can be loaded into a random access memory (RAM) 803 according to a program stored in a read only memory (ROM) 802 or a program from a storage section 808 Instead, various appropriate actions and processes are performed. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801 , the ROM 802 , and the RAM 803 are connected to each other through a bus 804 . An input/output (I/O) interface 805 is also connected to bus 804 .

The following components are connected to the I/O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc. ; and a communication section 809 including a network interface card such as a LAN card, a modem, and the like. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs in accordance with the disclosed embodiments of the present invention. For example, embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 809, and/or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-described functions defined in the system of the present application are executed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules can also be set in the processor, for example, it can be described as: a processor includes an image acquisition module, a feature map processing module, and a target detection module. Among them, the names of these modules do not constitute a limitation of the module itself in some cases, for example, the image acquisition module can also be described as "a module for acquiring 2D images and 3D depth images of detection targets".

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by a device, the device includes: acquiring a 2D image and a 3D depth image of the detection target; using the 3D depth image as a masking data, processing the first feature map generated by the 2D image to obtain a second feature map; inputting the second feature map into a target detection network to obtain category information and position information corresponding to the detection target.

According to the technical scheme of the embodiment of the present invention, a 2D image and a 3D depth image of the detection target are collected; the 3D depth image is used as masking data, and the first feature map generated from the 2D image is processed to obtain a second feature map; The second feature map is input to the target detection network to obtain category information and location information corresponding to the detected target. The identification of the detection target and its respective planes can be improved, the detection accuracy can be improved, the additional calculation burden is not increased, and the detection cost can be saved.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (8)

1. A method of object detection, comprising:

collecting a 2D image and a 3D depth image of a detection target;

taking the 3D depth image as masking data, and processing a first feature map generated by the 2D image to obtain a second feature map;

and inputting the second characteristic diagram into a target detection network to obtain the category information and the position information corresponding to the detection target.

2. The method of claim 1, wherein the step of processing the first feature map generated from the 2D image using the 3D depth image as mask data to obtain a second feature map comprises:

performing N-level convolution and pooling on the 2D image to obtain N first feature maps;

performing N-level pooling on the 3D depth image to obtain N pooled 3D depth images, wherein the pooled 3D depth images correspond to the first feature map one by one, and N is a positive integer;

and for each first feature map, using the corresponding pooled 3D depth image as masking data, and performing preset processing on the feature in the first feature map to obtain a second feature map.

3. The method of claim 1, wherein the predetermined processing comprises one of dot multiplication, addition and combination.

4. An object detection device, comprising:

the image acquisition module is used for acquiring a 2D image and a 3D depth image of a detection target;

the feature map processing module is used for processing a first feature map generated by the 2D image by taking the 3D depth image as masking data to obtain a second feature map;

and the target detection module is used for inputting the second characteristic diagram into a target detection network to obtain the category information and the position information corresponding to the detection target.

5. The apparatus of claim 4, wherein the feature map processing module is further configured to:

performing N-level convolution and pooling on the 2D image to obtain N first feature maps;

performing N-level pooling on the 3D depth image to obtain N pooled 3D depth images, wherein the pooled 3D depth images correspond to the first feature map one by one, and N is a positive integer;

and for each first feature map, using the corresponding pooled 3D depth image as masking data, and performing preset processing on the feature in the first feature map to obtain a second feature map.

6. The apparatus of claim 4, wherein the predetermined process comprises one of dot multiplication, addition and combination.

7. An electronic device, comprising:

one or more processors;

a memory for storing one or more programs,

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-3.

8. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-3.

Images