CN110632608A - A target detection method and device based on laser point cloud - Google Patents

Abstract

The invention discloses a laser point cloud-based target detection method and device, and relates to the technical field of computers. A specific implementation of the method includes: rasterizing the collected laser point cloud data, and extracting features from each grid to obtain three-dimensional lattice data; performing three-dimensional convolution and three-dimensional downsampling on the three-dimensional lattice data, To obtain a three-dimensional feature map; corresponding to each position of the three-dimensional feature map, generate multiple three-dimensional detection frames with the same height, and select a candidate three-dimensional detection frame from the three-dimensional detection frame; for each candidate three-dimensional detection frame The corresponding three-dimensional feature Figure, perform ROI downsampling on the length and width dimensions to obtain feature maps of the same size corresponding to each three-dimensional detection frame; perform classification and regression processing according to the feature map of the same size corresponding to each three-dimensional detection frame to determine the category and location information. This embodiment can not rely on the calibration between the lidar and the camera, and the accuracy of the detection result is high.

Description

A target detection method and device based on laser point cloud

technical field

The present invention relates to the field of computer technology, in particular to a laser point cloud-based target detection method and device.

Background technique

Using target detection technology, the smallest three-dimensional cuboid frame that can envelop the detection target can be determined in the laser point cloud in three-dimensional space. Taking vehicle detection in the field of automatic driving as an example, a vehicle corresponds to a three-dimensional cuboid frame. At present, the target detection effect is better based on the result of image-based target detection, but it is difficult to obtain accurate position information only by relying on images. Therefore, it is necessary to calibrate the lidar and camera, and then map the detected targets in the image to laser points. On the cloud, and then make decisions based on the location information on the point cloud. If the calibration is not accurate, the position of the target mapped to the laser point cloud will be inaccurate, thus affecting the accuracy of the detection results.

In the course of realizing the present invention, the inventor finds that there are at least the following problems in the prior art:

Existing methods rely on the calibration of lidar and camera, and the accuracy of detection results is poor.

Contents of the invention

In view of this, the embodiments of the present invention provide a laser point cloud-based target detection method and device, which can be independent of the calibration between the laser radar and the camera, and the accuracy of the detection result is high.

To achieve the above object, according to an aspect of the embodiments of the present invention, a laser point cloud based object detection method and device are provided.

A target detection method based on laser point cloud, comprising: rasterizing the collected laser point cloud data, and extracting features from each grid to obtain three-dimensional lattice data; performing three-dimensional convolution on the three-dimensional lattice data Product and three-dimensional downsampling to obtain a three-dimensional feature map; corresponding to each position of the three-dimensional feature map, generate a plurality of three-dimensional detection frames with the same height, and select a candidate three-dimensional detection frame from the three-dimensional detection frame; For the three-dimensional feature map corresponding to each candidate three-dimensional detection frame, perform ROI (region of interest) downsampling on the length and width dimensions to obtain the same size feature map corresponding to each three-dimensional detection frame; according to the corresponding three-dimensional detection frame Classification and regression processing are performed on the feature maps of the same size to determine the category and location information of the detection target.

Optionally, corresponding to each position of the 3D feature map, generating a plurality of 3D detection frames with the same height, and selecting a candidate 3D detection frame from the 3D detection frames includes: corresponding to the 3D feature Multiple 3D detection frames with the same height are generated for each position in the graph, and the probability that each 3D detection frame belongs to the foreground is determined; the non-maximum value suppression algorithm is used to deduplicate each 3D detection frame, and each 3D detection frame after deduplication Select a preset number of 3D detection frames with the highest probability belonging to the foreground as candidate 3D detection frames.

Optionally, the probability that the three-dimensional detection frame belongs to the foreground is determined by the following method: mapping the three-dimensional detection frame onto a two-dimensional plane to obtain a first two-dimensional detection frame corresponding to the three-dimensional detection frame; mapping a preset cuboid On the two-dimensional plane, a rectangular frame corresponding to the preset cuboid is obtained, and the preset cuboid is a pre-marked detection target sample; according to the intersection of the first two-dimensional detection frame and the rectangular frame And compare, determine the probability that the 3D detection frame belongs to the foreground.

Optionally, for the 3D feature map corresponding to each candidate 3D detection frame, ROI downsampling is performed on the length and width dimensions to obtain the same size feature map corresponding to each 3D detection frame, including: for each candidate 3D The three-dimensional feature map corresponding to the detection frame is based on the length and width dimensions, and the feature map corresponding to the four second two-dimensional detection frames is obtained; the ROI downsampling is performed on the feature map corresponding to each second two-dimensional detection frame, and each second two-dimensional detection frame is obtained. The feature maps of the same size corresponding to the three-dimensional detection frame; the feature maps of the same size corresponding to each second two-dimensional detection frame are combined into feature maps of the same size corresponding to each three-dimensional detection frame according to the corresponding candidate three-dimensional detection frame.

According to another aspect of the embodiments of the present invention, an object detection device based on a laser point cloud is provided.

A target detection device based on laser point cloud, including: a point cloud data processing module, used to rasterize the collected laser point cloud data, and extract features from each grid to obtain three-dimensional lattice data; feature map The generating module is used to perform three-dimensional convolution and three-dimensional down-sampling on the three-dimensional lattice data to obtain a three-dimensional feature map; the candidate frame generating module is used to generate corresponding to each position of the three-dimensional feature map with the same height A plurality of three-dimensional detection frames, and selecting a candidate three-dimensional detection frame from the three-dimensional detection frame; ROI down-sampling module, used to perform ROI reduction on the length and width dimensions of the three-dimensional feature map corresponding to each candidate three-dimensional detection frame Sampling to obtain feature maps of the same size corresponding to each three-dimensional detection frame; the detection module is used to perform classification and regression processing according to the feature map of the same size corresponding to each three-dimensional detection frame to determine the category and position information of the detection target.

Optionally, the candidate frame generating module is further configured to: generate a plurality of three-dimensional detection frames with the same height corresponding to each position of the three-dimensional feature map, and determine the probability that each three-dimensional detection frame belongs to the foreground; The value suppression algorithm deduplicates each 3D detection frame, and selects a preset number of 3D detection frames with the highest probability of belonging to the foreground from the deduplicated 3D detection frames as candidate 3D detection frames.

Optionally, the candidate frame generation module includes a foreground determination submodule, configured to: map the 3D detection frame onto a 2D plane to obtain a first 2D detection frame corresponding to the 3D detection frame; Assuming that a cuboid is mapped onto the two-dimensional plane, a rectangular frame corresponding to the preset cuboid is obtained, and the preset cuboid is a pre-marked detection target sample; according to the first two-dimensional detection frame and the rectangle The intersection and union ratio of the frame determines the probability that the 3D detection frame belongs to the foreground.

Optionally, the detection module is further configured to: for the three-dimensional feature map corresponding to each candidate three-dimensional detection frame, obtain the feature map corresponding to four second two-dimensional detection frames based on the length and width dimensions; The feature map corresponding to the two-dimensional detection frame is subjected to ROI down-sampling to obtain the same size feature map corresponding to each second two-dimensional detection frame; the same size feature map corresponding to each second two-dimensional detection frame, according to the corresponding candidate three-dimensional detection frame, Combined into feature maps of the same size corresponding to each 3D detection frame.

Optionally, the target detection device based on laser point cloud also includes a training module, configured to: train the feature map generation module, the candidate frame generation module, the ROI downsampling module and the detection module.

According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

An electronic device, comprising: one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more A plurality of processors realize the object detection method based on the laser point cloud provided by the present invention.

According to yet another aspect of the embodiments of the present invention, a computer-readable medium is provided.

A computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, the object detection method based on the laser point cloud provided by the present invention is implemented.

An embodiment of the above invention has the following advantages or beneficial effects: rasterize the collected laser point cloud data, and extract features from each grid to obtain three-dimensional lattice data; perform three-dimensional convolution on the three-dimensional lattice data and three-dimensional downsampling to obtain a three-dimensional feature map; corresponding to each position of the three-dimensional feature map, multiple three-dimensional detection frames with the same height are generated, and candidate three-dimensional detection frames are selected from the three-dimensional detection frames; for each candidate three-dimensional detection frame The three-dimensional feature map corresponding to the frame, ROI downsampling is performed on the length and width dimensions to obtain the same size feature map corresponding to each three-dimensional detection frame; classification and regression processing are performed according to the same size feature map corresponding to each three-dimensional detection frame to determine The category and location information of the detected target. By directly processing the collected three-dimensional laser point cloud data, the present invention can not rely on the calibration between the laser radar and the camera, and the accuracy of the detection result is high.

The further effects of the above-mentioned non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used to better understand the present invention, and do not constitute improper limitations to the present invention. in:

Fig. 1 is a schematic diagram of the main steps of a target detection method based on a laser point cloud according to an embodiment of the present invention;

2 is a schematic diagram of the composition of a target detection model according to an embodiment of the present invention;

3 is a schematic diagram of main modules of a target detection device based on a laser point cloud according to an embodiment of the present invention;

FIG. 4 is an exemplary system architecture diagram to which an embodiment of the present invention can be applied;

Fig. 5 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a schematic diagram of main steps of a target detection method based on a laser point cloud according to an embodiment of the present invention.

As shown in FIG. 1 , the object detection method based on the laser point cloud in the embodiment of the present invention mainly includes the following steps S101 to S105.

Step S101: rasterizing the collected laser point cloud data, and extracting features from each raster to obtain 3D point matrix data.

Rasterizing the laser point cloud data may be to rasterize the laser point cloud data within a preset range, the preset range is related to the task of target detection, and its value can be set according to experience. For example, in the vehicle detection task in unmanned driving, laser point cloud data within the range of 40 meters forward, 20 meters left and right, and a height of 0 to 10 meters can be selected.

The three-dimensional lattice data includes four eigenvalues corresponding to each grid. The four eigenvalues are the maximum height of the point cloud in the raster, the reflection intensity of the point cloud with the maximum height, the number of point clouds in the raster, and an indicator value indicating whether there is a point in the raster (1 means a point in the raster , 0 means no points in the grid). Whether the current grid is a part of the detection target can be well represented by the four eigenvalues.

Step S102: Perform 3D convolution and 3D downsampling on the 3D lattice data to obtain a 3D feature map.

Step S103: Corresponding to each position of the 3D feature map, generate multiple 3D detection frames with the same height, and select candidate 3D detection frames from the 3D detection frames.

Step S103 may specifically include: generating a plurality of three-dimensional detection frames corresponding to each position of the three-dimensional feature map, so as to determine the probability, position information and scale information of each three-dimensional detection frame belonging to the foreground, and the scale information of each three-dimensional detection frame includes the same height; The non-maximum value suppression algorithm deduplicates each 3D detection frame, and selects a preset number of 3D detection frames with the highest probability belonging to the foreground from each 3D detection frame after deduplication, as candidate 3D detection frames.

Wherein, the probability that the 3D detection frame belongs to the foreground is determined by the following method: mapping the 3D detection frame onto a 2D plane to obtain a first 2D detection frame corresponding to the 3D detection frame; mapping a preset cuboid onto a 2D plane , to obtain a rectangular frame corresponding to a preset cuboid, which is a pre-marked detection target sample; according to the intersection ratio between the first two-dimensional detection frame and the rectangular frame, determine the probability that the three-dimensional detection frame belongs to the foreground .

Step S104: For the 3D feature map corresponding to each candidate 3D detection frame, ROI downsampling is performed in the length and width dimensions to obtain feature maps of the same size corresponding to each 3D detection frame.

Step S104 may specifically include: for the three-dimensional feature map corresponding to each candidate three-dimensional detection frame, based on the length and width dimensions, obtain the feature map corresponding to four second two-dimensional detection frames; for the feature map corresponding to each second two-dimensional detection frame The image is subjected to ROI downsampling to obtain feature maps of the same size corresponding to each second two-dimensional detection frame; the feature maps of the same size corresponding to each second two-dimensional detection frame are combined into corresponding three-dimensional detection frames according to the corresponding candidate three-dimensional detection frame The same size feature map of the box.

By performing ROI downsampling in the length and width dimensions, ROI downsampling can be used to process 3D feature maps, which in turn makes it possible to directly process the collected 3D laser point cloud data without relying on lidar and cameras. Calibration between.

Step S105: Perform classification and regression processing according to the feature maps of the same size corresponding to each three-dimensional detection frame to determine the category and location information of the detection target.

Specifically, two consecutive three-dimensional convolution processes are performed on the feature maps of the same size corresponding to each three-dimensional detection frame, and then classification and regression processing are performed on the feature map of the same size corresponding to each three-dimensional detection frame after the three-dimensional convolution process, To determine the category and location information of the detected target.

Taking the vehicle detection in the field of automatic driving as an example, the object detection method based on the laser point cloud according to the embodiment of the present invention will be introduced in detail below. The object detection method based on the laser point cloud in the embodiment of the present invention is not limited to detecting vehicles, but can also be used to detect other objects such as pedestrians.

The embodiments of the present invention directly process the collected three-dimensional laser point cloud data to detect the vehicles therein without relying on the calibration between the laser radar and the camera, so that the vehicle detection results are more accurate.

First, rasterize the laser point cloud data collected in the three-dimensional space, specifically, select the laser point cloud data within the range of 40 meters forward, 20 meters left and right, and a height of 0 to 10 meters, and for the selected The laser point cloud data in the above range is rasterized, in which every 0.1 meters in the forward direction and left and right corresponds to a grid, and every 0.4 meters in height corresponds to a grid. Therefore, the laser point cloud data is divided into 400*400*25 three-dimensional grid, the selected laser point cloud data in the above range falls into one of these grids, and extracts 4 features for each grid, which are the maximum height of the point cloud in the grid and the point with the maximum height The reflection intensity of the cloud, the number of point clouds in the grid, and the indication value indicating whether there are points in the grid. The indication value can be 0 or 1. Among them, if there are points in the grid, the indication value is 1, and if there are no points, the indication value is 0, where the points are the points that make up the laser point cloud.

Rasterize the laser point cloud data, and extract the above four features for each grid, so as to obtain 3D lattice data of 400*400*25, and the position of each grid in the 3D lattice data has 4 channels value, which is the value of the above four features. By extracting these 4 features, it can well represent whether the current grid is a part of the vehicle. It should be noted that if the detection target is a pedestrian or other object, the range of the laser point cloud data selected when rasterizing the laser point cloud data mentioned above needs to be adaptively adjusted according to the size of the detection target.

In the embodiment of the present invention, a target detection model can be constructed based on a Faster RCNN (Faster RCNN based on image region) framework to perform target detection.

First, based on the network structure of convolutional neural networks such as the VGG16 network, a 3D convolutional neural network can be constructed to perform 3D convolution and 3D downsampling on 3D lattice data to obtain a 3D feature map. For example, keep the number of convolution kernels, the size of the convolution kernel, and the step size of the VGG16 network unchanged, increase the dimension of the convolution kernel, and change the original two-dimensional convolution kernel into a three-dimensional convolution kernel, so that the two-dimensional convolution The accumulation layer becomes a three-dimensional convolutional layer, which can realize convolution on three-dimensional volume. The present invention can also construct the above-mentioned three-dimensional convolutional neural network based on the network structure of other convolutional neural networks (such as GoogleNet MobileNet, etc.).

Specifically, all the convolutional layers and downsampling layers of the VGG16 network except the last downsampling layer are retained. The reason why the last downsampling layer is removed is because the data will lose some useful information for vehicle detection tasks after being processed by this layer. information. According to the above introduction, the original two-dimensional convolutional layer of the VGG network is changed to a three-dimensional convolutional layer. Correspondingly, the two-dimensional parameters become three-dimensional parameters. In addition, the downsampling layer is changed to three-dimensional downsampling, but Downsampling is only performed in the forward and left and right directions. Since there are fewer dimensions in the height direction (in this case, the height is only 25), in order to avoid losing important information through downsampling, no downsampling is performed on the height. After three-dimensional convolution and three-dimensional downsampling, a 25*25*25 three-dimensional feature map is obtained.

Then, a classification layer (also called the first classification layer) and a regression layer (also called the first regression layer) are connected behind the last layer of the three-dimensional convolutional neural network (relu layer, ie, the activation layer). layer is a fully connected layer. Through the first classification layer and the first regression layer, multiple three-dimensional detection frames with the same height are generated corresponding to each position of the three-dimensional feature map. Each position of the 3D feature map corresponds to a 3D detection frame. The process of generating the 3D detection frame is the process of using the first classification layer and the first regression layer to determine the probability, position information and scale information of each 3D detection frame belonging to the foreground. .

Wherein, the first classification layer is used to determine whether the 3D detection frame belongs to the foreground or the background, and the first regression layer is used to determine the position information of the 3D detection frame. It should be noted that, in the training phase of the target detection model in the embodiment of the present invention, the first regression layer learns 6 values of the three-dimensional detection frame, which are the coordinates (x, y, z) of the center point of the three-dimensional detection frame and the length, width and height of the 3D detection box.

The method of generating the 3D detection frame is as follows. First, multiple 3D detection frames are generated for each position of the 25*25*25 3D feature map. Specifically, the first classification layer is used to determine the probability that each 3D detection frame belongs to the foreground, and the first regression layer is used to determine the position information and scale information of each 3D detection frame. In order to determine whether the 3D detection frame belongs to the foreground or the background, the 3D detection frame is mapped onto a 2D plane to obtain a 2D detection frame (ie, the first 2D detection frame). Then, the three-dimensional cuboid that may contain the vehicle is mapped to the two-dimensional plane to obtain a rectangular box, which is a pre-marked cuboid that contains the vehicle. When judging whether the 3D detection frame belongs to the foreground or the background, calculate the IOU of the two according to the 2D detection frame obtained by mapping the 3D detection frame to a 2D plane, and the rectangular box obtained by mapping a cuboid that may contain a vehicle to a 2D plane. (Intersection-over-union ratio), if the IOU is greater than 0.7, the probability of belonging to the foreground is greater, and it is considered to be the foreground, and if it is less than 0.5, the probability of belonging to the background is greater, and it is considered to be the background, and the rest of the values are ignored. It should be noted that the side of the above-mentioned rectangular frame may have a rotation angle with the x-direction (or y-reverse) of the image. When training the first regression layer, the first regression layer needs to learn 6 A value, set to include the coordinates of the center point, length, width and height of the smallest cuboid parallel to the x-direction and y-direction of the above cuboid.

For each position on the 3D feature map output by the 3D convolutional neural network, generate 4 3D detection boxes, where (length, width, height) are equal to (39,16,4), (16,39,4) respectively , (10,6,4) and (6,10,4) are four kinds in total, and the unit is the number of points, so the three-dimensional detection frame of this scale is in line with the size of the vehicle. If the detection target is a pedestrian or other object, a three-dimensional detection frame with a scale consistent with the size of the pedestrian or other object is generated.

Candidate 3D detection boxes are selected from the 3D detection boxes. Specifically, a 3D detection frame with high confidence is selected from the 3D detection frames belonging to the foreground, that is, a preset number of 3D detection frames output by the first classification layer with the highest probability of belonging to the foreground are selected as candidate 3D detection frames. In the training phase of the target detection model, 12,000 3D detection frames are selected, and 6,000 3D detection frames are selected in the test phase, and then the non-maximum value suppression algorithm is used to detect objects with high overlap (for example, the overlap rate is higher than a preset threshold). Only one 3D detection frame with the highest confidence is retained in the 3D detection frame, and the first 2000 3D detection frames after suppression can be selected as candidate 3D detection frames in the training phase, and the top 300 3D detection frames after suppression can be selected as candidates in the test phase 3D detection box.

Use a ROI downsampling layer, according to the 3D feature map output by the last layer of VGG16, and the candidate 3D detection frame obtained above, downsample the candidate 3D detection frames of different sizes to obtain the same size features corresponding to each 3D detection frame picture.

The difference between the ROI downsampling layer and the traditional downsampling layer is that the ROI downsampling layer can downsample the feature maps corresponding to the detection frames of different scales to feature maps of the same size. Since the heights of the candidate 3D detection frames are all 4 (according to this example Above, the height is a grid every 0.4 meters, 4 represents 1.6 meters, if the detection target is a pedestrian or other object, the height is other values, the specific value is related to the size of the detection target), therefore, only the length and width ROI downsampling is performed in each dimension, so that all candidate 3D detection frames have feature maps with the same length and width (same height), so that the feature maps of the same size of each 3D detection frame have length, width, and height, and correspond to the VGG16 network. 512-dimensional channels. After the ROI downsampling layer, two three-dimensional convolutional layers are connected. The number of convolutional kernels is 128, the size of the convolutional kernels is 1*1*1, and the step size is 1. The two three-dimensional convolutional layers are used to extract features.

After the two three-dimensional convolutional layers, a classification layer (also called the second classification layer) and a regression layer (also called the second regression layer) are connected, and the second classification layer is used to determine the category of the candidate three-dimensional detection frame. The binary regression layer is used to determine the specific position information of the candidate 3D detection frame. There are two types of candidate 3D detection frames, namely vehicle or background. The specific position information of the candidate 3D detection frame is the coordinates of the 8 vertices of the 3D frame, with a total of 24 values.

The target detection model of the embodiment of the present invention may be shown in FIG. 2 .

When training the network of the target detection model as shown in Figure 2, A samples (A=128 or 256) can be arbitrarily selected from 2000 candidate boxes, among which there are a certain proportion of negative samples, that is, the background, using these selected samples Gradient descent is performed on the network, and the learning of the network adopts the backpropagation algorithm and the stochastic gradient descent method. Specifically, the true value labels of the training samples are marked before training. For the first classification layer and the first regression layer, according to the marked true value labels (that is, foreground or background, and the position information of the three-dimensional detection frame) during each training ), and the output results of the first classification layer and the first regression layer, calculate the classification cost and regression cost, for the second classification layer and the second regression layer, according to the marked true value label (ie vehicle or background, and vehicle's Specific location), and the output results of the second classification layer and the second regression layer, calculate the classification cost and regression cost, continuously reduce the total Loss value (total Loss is the total cost, including classification cost and regression cost), and finally get the output comparison The output values of the accurate classification layer (the first classification layer and the second classification layer) and the regression layer (the first regression layer and the second regression layer), the gradient descent continuously moves the Loss value to the opposite direction of the gradient corresponding to the current point, To reduce Loss, stochastic gradient descent only updates the gradient calculated by one training sample each time, and the backpropagation algorithm is used to find the gradient.

In the embodiment of the present invention, the OHEM (Online hard example mining) method can also be used to train the network. Unlike the random selection of A samples in the above training process, the embodiment of the present invention uses the OHEM method, using each candidate frame as a sample, Calculate its total Loss value, and sort the total Loss value of each sample from large to small, select A samples (A=128 or 256) with the largest total Loss value, use these selected samples to perform gradient descent on the network, and also Generally, the learning of the network adopts the backpropagation algorithm and the stochastic gradient descent method, which will not be repeated here. In this way, some samples that are difficult to learn can also be well learned, making the result of vehicle detection more accurate.

Fig. 3 is a schematic diagram of main modules of an object detection device based on a laser point cloud according to an embodiment of the present invention.

The object detection device 300 based on the laser point cloud in the embodiment of the present invention mainly includes a point cloud data processing module 301 , a feature map generation module 302 , a candidate frame generation module 303 , an ROI downsampling module 304 , and a detection module 305 .

The point cloud data processing module 301 is used to rasterize the collected laser point cloud data, and extract features for each grid to obtain three-dimensional lattice data, and the three-dimensional lattice data includes four eigenvalues corresponding to each grid (obtained by extracting four features for each grid), the four feature values of each grid are: the maximum height of the point cloud in the grid, the reflection intensity of the point cloud with the maximum height, and The number of point clouds, and an indicator value indicating whether points are in this raster.

The feature map generation module 302 is used to perform three-dimensional convolution and three-dimensional down-sampling on the three-dimensional lattice data to obtain a three-dimensional feature map.

The candidate frame generation module 303 is used to generate multiple 3D detection frames with the same height corresponding to each position of the 3D feature map, and select candidate 3D detection frames from the 3D detection frames.

The candidate frame generating module 303 can be specifically used to: generate a plurality of three-dimensional detection frames corresponding to each position of the three-dimensional feature map, so as to determine the probability, position information and scale information of each three-dimensional detection frame belonging to the foreground, and the scale information of each three-dimensional detection frame includes The same height; use the non-maximum value suppression algorithm to deduplicate each 3D detection frame, and select a preset number of 3D detection frames with the highest probability belonging to the foreground from each 3D detection frame after deduplication, as a candidate 3D detection frame.

The ROI downsampling module 304 is used to perform ROI downsampling on the length and width dimensions of the 3D feature map corresponding to each candidate 3D detection frame, so as to obtain feature maps of the same size corresponding to each 3D detection frame.

The detection module 305 is used to perform classification and regression processing according to the feature maps of the same size corresponding to each three-dimensional detection frame, so as to determine the category and location information of the detection target.

The candidate frame generation module 303 may include a foreground determination submodule, configured to: map the three-dimensional detection frame onto a two-dimensional plane to obtain a first two-dimensional detection frame corresponding to the three-dimensional detection frame; map a preset cuboid onto a two-dimensional plane , to obtain a rectangular frame corresponding to a preset cuboid, which is a pre-marked detection target sample; according to the intersection ratio between the first two-dimensional detection frame and the rectangular frame, determine the probability that the three-dimensional detection frame belongs to the foreground.

The detection module 305 can specifically be used to: for the three-dimensional feature map corresponding to each candidate three-dimensional detection frame, based on the length and width dimensions, obtain the feature map corresponding to four second two-dimensional detection frames; The feature map of the ROI is down-sampled to obtain the same size feature map corresponding to each second two-dimensional detection frame; the same size feature map corresponding to each second two-dimensional detection frame is combined according to the corresponding candidate three-dimensional detection frame. Same-size feature maps of 3D detection boxes.

The object detection device 300 based on the laser point cloud can also include a training module for training the feature map generation module 302 , the candidate frame generation module 303 , the ROI downsampling module 304 and the detection module 305 through the OHEM training method.

The target detection device 300 based on the laser point cloud of the present invention can be realized based on the target detection model constructed above, specifically, after the three-dimensional lattice data is obtained by the point cloud data processing module 301, the three-dimensional lattice data can be used as the target The input of the detection model can realize the corresponding function of the feature map generation module 302 through the three-dimensional convolutional neural network in the target detection model. The function of the candidate frame generation module 303 to generate multiple three-dimensional detection frames with the same height can be realized through the first classification layer and the first regression layer, and then select the preset number of three-dimensional detection frames output by the classification layer with the highest probability of belonging to the foreground , as the candidate 3D detection frame, so as to realize the function of selecting the candidate 3D detection frame of the candidate frame generation module 303 . The corresponding functions of the ROI down-sampling module 304 are realized through the ROI down-sampling layer, or the corresponding functions of the ROI down-sampling module 304 are realized through the combination of the ROI down-sampling layer and two three-dimensional convolution layers. The corresponding functions of the detection module 305 are realized through the second classification layer and the second regression layer. Therefore, the above-mentioned training module can also be used to train each layer of the object detection model of the embodiment of the present invention. Since the OHEM training method has been introduced in detail above, it will not be repeated here.

In addition, the specific implementation content of the target detection device based on laser point cloud in the embodiment of the present invention has been described in detail in the above-mentioned target detection method based on laser point cloud, so the repeated content will not be described here.

FIG. 4 shows an exemplary system architecture 400 that can be applied to a laser point cloud-based target detection method or a laser point cloud-based target detection device according to an embodiment of the present invention.

As shown in FIG. 4 , the system architecture 400 may include terminal devices 401 , 402 , 403 , a network 404 and a server 405 . The network 404 is used as a medium for providing communication links between the terminal devices 401 , 402 , 403 and the server 405 . Network 404 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.

Users can use terminal devices 401 , 402 , 403 to interact with server 405 via network 404 to receive or send messages and the like. Various communication client applications can be installed on the terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, and the like.

The terminal devices 401, 402, 403 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers and the like.

The server 405 may be a server that provides various services, such as a background management server that provides support for shopping websites browsed by users using the terminal devices 401 , 402 , and 403 . The background management server can analyze and process the received data such as the target information query request, and feed back the processing results (such as target information) to the terminal device.

It should be noted that the target detection method based on the laser point cloud provided by the embodiment of the present invention can be executed by the server 405 or the terminal equipment 401, 402, 403, and correspondingly, the target detection device based on the laser point cloud can be set on the server 405 Or in the terminal equipment 401, 402, 403.

It should be understood that the numbers of terminal devices, networks and servers in Fig. 4 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

Referring now to FIG. 5 , it shows a schematic structural diagram of a computer system 500 suitable for implementing a terminal device or a server according to an embodiment of the present application. The terminal device or server shown in FIG. 5 is only an example, and should not limit the functions and application scope of this embodiment of the present application.

As shown in FIG. 5 , a computer system 500 includes a central processing unit (CPU) 501 that can be programmed according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage section 508 into a random-access memory (RAM) 503 Instead, various appropriate actions and processes are performed. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501 , ROM 502 , and RAM 503 are connected to each other through a bus 504 . An input/output (I/O) interface 505 is also connected to the bus 504 .

The following components are connected to the I/O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 508 including a hard disk, etc. and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 510 as necessary so that a computer program read therefrom is installed into the storage section 508 as necessary.

In particular, according to the disclosed embodiments of the present invention, the processes described above with reference to the schematic diagrams of main steps can be implemented as computer software programs. For example, the disclosed embodiments of the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for executing the method shown in the schematic diagram of main steps. In such an embodiment, the computer program may be downloaded and installed from a network via communication portion 509 and/or installed from removable media 511 . When this computer program is executed by a central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are performed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program codes are carried. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. . Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The schematic diagrams and block diagrams of main steps in the drawings illustrate the architecture, functions and operations of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the principal step diagram or block diagram may represent a module, program segment, or portion of code that contains one or more logic components for implementing the specified Executable instructions for a function. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It is also to be noted that each block of the block diagrams or major step illustrations, and combinations of blocks in the block diagrams or major step illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or Implementation can be by a combination of special purpose hardware and computer instructions.

The modules involved in the embodiments described in the present invention may be implemented by software or by hardware. The described modules can also be set in a processor, for example, can be described as: a processor includes a point cloud data processing module 301, a feature map generation module 302, a candidate frame generation module 303, an ROI downsampling module 304, a detection module 305. Wherein, the names of these modules do not constitute a limitation of the module itself in some cases, for example, the point cloud data processing module 301 can also be described as "for rasterizing the collected laser point cloud data, and A module for extracting features from each grid to obtain 3D lattice data".

As another aspect, the present invention also provides a computer-readable medium. The computer-readable medium may be contained in the device described in the above embodiments, or it may exist independently without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by one of the devices, the device includes: rasterizing the collected laser point cloud data, and extracting each raster features to obtain 3D lattice data; perform 3D convolution and 3D downsampling on the 3D lattice data to obtain a 3D feature map; corresponding to each position of the 3D feature map, generate multiple 3D images with the same height detection frame, and select a candidate three-dimensional detection frame from the three-dimensional detection frame; for the three-dimensional feature map corresponding to each candidate three-dimensional detection frame, ROI downsampling is performed on the length and width dimensions to obtain the corresponding three-dimensional detection frame The feature map of the same size; performing classification and regression processing according to the feature map of the same size corresponding to each three-dimensional detection frame, so as to determine the category and position information of the detection target.

According to the technical solution of the embodiment of the present invention, the collected laser point cloud data is rasterized, and features are extracted from each grid to obtain three-dimensional lattice data; three-dimensional convolution and three-dimensional downsampling are performed on the three-dimensional lattice data, To obtain a three-dimensional feature map; corresponding to each position of the three-dimensional feature map, generate a plurality of three-dimensional detection frames with the same height, and select a candidate three-dimensional detection frame from the three-dimensional detection frame; for each candidate three-dimensional detection frame corresponding to the three-dimensional feature Figure, perform ROI down-sampling on the length and width dimensions to obtain feature maps of the same size corresponding to each 3D detection frame; perform classification and regression processing according to the feature map of the same size corresponding to each 3D detection frame to determine the category and location information. It can not rely on the calibration between the lidar and the camera, and the accuracy of the detection result is high.

The above specific implementation methods do not constitute a limitation to the protection scope of the present invention. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (11)

1. A target detection method based on laser point cloud is characterized by comprising the following steps:

rasterizing the collected laser point cloud data, and extracting features of each grid to obtain three-dimensional dot matrix data;

performing three-dimensional convolution and three-dimensional down-sampling on the three-dimensional lattice data to obtain a three-dimensional characteristic diagram;

generating a plurality of three-dimensional detection frames with the same height corresponding to each position of the three-dimensional feature map, and selecting candidate three-dimensional detection frames from the three-dimensional detection frames;

performing ROI (region of interest) downsampling on the length dimension and the width dimension of the three-dimensional feature map corresponding to each candidate three-dimensional detection frame to obtain the feature maps with the same size corresponding to each three-dimensional detection frame;

and carrying out classification and regression processing according to the feature maps with the same size corresponding to the three-dimensional detection frames so as to determine the category and the position information of the detection target.

2. The method according to claim 1, wherein the step of generating a plurality of three-dimensional detection frames having the same height for each position of the three-dimensional feature map and selecting a candidate three-dimensional detection frame from the three-dimensional detection frames comprises:

generating a plurality of three-dimensional detection frames with the same height corresponding to each position of the three-dimensional characteristic diagram, and determining the probability that each three-dimensional detection frame belongs to the foreground;

and removing the duplication of the three-dimensional detection frames by using a non-maximum suppression algorithm, and selecting a preset number of three-dimensional detection frames with the highest probability of belonging to the foreground from the three-dimensional detection frames after the duplication removal as candidate three-dimensional detection frames.

3. The method of claim 2, wherein the probability that the three-dimensional detection box belongs to the foreground is determined by:

mapping the three-dimensional detection frame to a two-dimensional plane to obtain a first two-dimensional detection frame corresponding to the three-dimensional detection frame;

mapping a preset cuboid onto the two-dimensional plane to obtain a rectangular frame corresponding to the preset cuboid, wherein the preset cuboid is a pre-marked detection target sample;

and determining the probability of the three-dimensional detection frame belonging to the foreground according to the intersection ratio of the first two-dimensional detection frame and the rectangular frame.

4. The method of claim 1, wherein the step of performing ROI downsampling on the length and width dimensions of the three-dimensional feature map corresponding to each candidate three-dimensional detection frame to obtain the same-size feature map corresponding to each three-dimensional detection frame comprises:

obtaining the three-dimensional feature maps corresponding to the four second two-dimensional detection frames based on the length dimension and the width dimension of the three-dimensional feature map corresponding to each candidate three-dimensional detection frame;

performing ROI (region of interest) downsampling on the feature map corresponding to each second two-dimensional detection frame to obtain the feature map with the same size corresponding to each second two-dimensional detection frame;

and combining the feature maps with the same size corresponding to the second two-dimensional detection frames into the feature maps with the same size corresponding to the three-dimensional detection frames according to the corresponding candidate three-dimensional detection frames.

5. A target detection device based on laser point cloud is characterized by comprising:

the point cloud data processing module is used for rasterizing the acquired laser point cloud data and extracting features of each grid to obtain three-dimensional dot matrix data;

the characteristic diagram generating module is used for carrying out three-dimensional convolution and three-dimensional down-sampling on the three-dimensional lattice data to obtain a three-dimensional characteristic diagram;

the candidate frame generation module is used for generating a plurality of three-dimensional detection frames with the same height corresponding to each position of the three-dimensional feature map and selecting a candidate three-dimensional detection frame from the three-dimensional detection frames;

the ROI down-sampling module is used for carrying out ROI down-sampling on the length dimension and the width dimension of the three-dimensional feature map corresponding to each candidate three-dimensional detection frame so as to obtain the feature map with the same size corresponding to each three-dimensional detection frame;

and the detection module is used for carrying out classification and regression processing according to the feature maps with the same size corresponding to the three-dimensional detection frames so as to determine the category and the position information of the detection target.

6. The apparatus of claim 5, wherein the candidate box generation module is further configured to:

generating a plurality of three-dimensional detection frames with the same height corresponding to each position of the three-dimensional characteristic diagram, and determining the probability that each three-dimensional detection frame belongs to the foreground;

and removing the duplication of the three-dimensional detection frames by using a non-maximum suppression algorithm, and selecting a preset number of three-dimensional detection frames with the highest probability of belonging to the foreground from the three-dimensional detection frames after the duplication removal as candidate three-dimensional detection frames.

7. The apparatus of claim 6, wherein the candidate block generation module comprises a foreground determination sub-module configured to:

mapping the three-dimensional detection frame to a two-dimensional plane to obtain a first two-dimensional detection frame corresponding to the three-dimensional detection frame;

mapping a preset cuboid onto the two-dimensional plane to obtain a rectangular frame corresponding to the preset cuboid, wherein the preset cuboid is a pre-marked detection target sample;

and determining the probability of the three-dimensional detection frame belonging to the foreground according to the intersection ratio of the first two-dimensional detection frame and the rectangular frame.

8. The apparatus of claim 5, wherein the detection module is further configured to:

obtaining the three-dimensional feature maps corresponding to the four second two-dimensional detection frames based on the length dimension and the width dimension of the three-dimensional feature map corresponding to each candidate three-dimensional detection frame;

performing ROI (region of interest) downsampling on the feature map corresponding to each second two-dimensional detection frame to obtain the feature map with the same size corresponding to each second two-dimensional detection frame;

and combining the feature maps with the same size corresponding to the second two-dimensional detection frames into the feature maps with the same size corresponding to the three-dimensional detection frames according to the corresponding candidate three-dimensional detection frames.

9. The apparatus of claim 5, wherein the laser point cloud based target detection apparatus further comprises a training module configured to:

training the feature map generation module, the candidate box generation module, the ROI downsampling module and the detection module by an OHEM training method.

10. An electronic device, comprising:

one or more processors;

a memory for storing one or more programs,

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method recited in any of claims 1-4.

11. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-4.

Images