CN107529650A - Network model construction and closed loop detection method, corresponding device and computer equipment - Google Patents

Abstract

The invention discloses a network model construction and closed-loop detection method, a corresponding device and computer equipment. The closed-loop detection method includes: inputting the currently captured real-scene image frame into the target network model constructed based on the network model construction method of the present invention to obtain the actual image features of the real-scene image frame; Match the image frame and the corresponding historical image features; based on the similarity value between the actual image feature and each historical image feature, determine the closed-loop detection result of the real-scene image frame. Using the above method, under the premise of ensuring the accuracy of closed-loop detection, the dimension of the image feature vector required in closed-loop detection can be effectively reduced, thereby shortening the calculation time of similarity calculation in closed-loop detection, so that the closed-loop detection can be better satisfied. Real-time requirements in testing.

Description

Construction of network model and closed-loop detection method, corresponding device and computer equipment

technical field

The invention relates to the technical field of machine learning, in particular to a network model construction and closed-loop detection method, a corresponding device and computer equipment.

Background technique

Image feature extraction is an important technical link in image processing in the field of computer vision. Traditional image feature extraction methods are very sensitive to illumination changes. When performing feature extraction on the same scene image captured under different lighting environments, different results often appear. This affects subsequent image processing performance.

On the basis of the above defects, technicians proposed a method for image feature extraction based on a deep learning model. Although the deep learning model can effectively avoid the influence of complex lighting on image features, the image output by the deep learning model proposed in the prior art The feature dimension of the feature is often high (for example, the image feature dimension output by the classic PlaceCNN convolutional network model is as high as 9126 dimensions), and the high-latitude image feature also greatly affects the calculation time of image processing and reduces the performance of image processing.

In addition, loop closure detection can be regarded as a common image processing problem in computer vision applications. During loop closure detection, if high-latitude image features are first extracted based on the existing deep learning model, the high-latitude image features will greatly affect the loop closure. The calculation time of subsequent similarity measures in the detection greatly affects the processing time of image processing (such as closed-loop detection) when performing subsequent image processing (such as closed-loop detection), so it is difficult to meet the requirements of real-time closed-loop detection.

Contents of the invention

Embodiments of the present invention provide network model construction and closed-loop detection methods, corresponding devices, and computer equipment, and realize network model construction. The constructed network model can output low-dimensional image features, and the output image features can realize real-time images. Closed loop detection.

In the first aspect, the embodiment of the present invention provides a method for constructing a network model, including:

Based on the obtained topology information and configuration parameter information, construct an initial network model, wherein the topology information includes at least one of the following: the number of layers of the convolutional layer, the number of layers of the pooling layer, and the number of layers of the fully connected layer , and the topological connection sequence between the layers; the configuration parameter information includes at least one of the following: the convolution step size and the convolution kernel size and quantity of each convolution layer, the pooling step size and the pooling step size of each pooling layer Optimize the window size and the number of neurons in each fully connected layer;

According to the obtained training and learning information, the initial network model is iteratively trained to obtain a target network model with a standard weight data set.

In a second aspect, an embodiment of the present invention provides a closed-loop detection method, including:

Inputting the currently captured real-scene image frame into a preset target network model to obtain actual image features of the real-scene image frame, and the target network model is determined based on the network model construction method provided in the embodiment of the first aspect of the present invention;

According to the set image frame selection rules, determine at least one image frame to be matched in the real-scene image frame, and acquire the historical image features of each image frame to be matched;

Based on the similarity value between the actual image feature and each historical image feature, the closed-loop detection result of the real-scene image frame is determined.

In a third aspect, an embodiment of the present invention provides an apparatus for constructing a network model, including:

An initial construction module, configured to construct an initial network model based on the obtained topology information and configuration parameter information, wherein the topology information includes at least one of the following: the number of layers of the convolutional layer, the number of layers of the pooling layer, The number of layers of the fully connected layer and the topological connection sequence between the layers; the configuration parameter information includes at least one of the following: the convolution step size of each convolution layer and the size and number of convolution kernels, the pool of each pooling layer Step size and pooling window size, and the number of neurons in each fully connected layer;

The target determination module is configured to iteratively train the initial network model according to the acquired training and learning information, and obtain a target network model with a standard weight data set.

In a fourth aspect, an embodiment of the present invention provides a closed-loop detection device, including:

A feature extraction module, configured to input the currently captured real-scene image frame into a preset target network model to obtain actual image features of the real-scene image frame, and the target network model is based on the network model provided by the embodiment of the third aspect of the present invention Build device determination;

An image selection module, configured to determine at least one image frame to be matched of the real-scene image frame according to the set image frame selection rule, and obtain historical image features of each image frame to be matched;

The detection and determination module is configured to determine the closed-loop detection result of the real-scene image frame based on the similarity value between the actual image feature and each historical image feature.

In a fifth aspect, an embodiment of the present invention provides a computer device, including:

one or more processors;

storage means for storing one or more programs;

The one or more programs are executed by the one or more processors, so that the one or more processors implement the network model construction method provided in the embodiment of the first aspect of the present invention.

In a sixth aspect, an embodiment of the present invention provides a computer device, including: a camera for capturing image frames, and also includes:

one or more processors;

storage means for storing one or more programs;

The one or more programs are executed by the one or more processors, so that the one or more processors implement the closed-loop detection method provided in the embodiment of the second aspect of the present invention.

In a seventh aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method for constructing a network model provided in the embodiment of the first aspect of the present invention is implemented;

In an eighth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the closed-loop detection method provided in the embodiment of the second aspect of the present invention is implemented.

In the network model construction and closed-loop detection method provided above, corresponding devices and computer equipment, in the network model construction method, the initial network model is first constructed based on the acquisition of preset topology information and configuration parameter information; then according to the obtained training Learning information, training to obtain the target network model with a standard weight data set. In the closed-loop detection method, first input the currently captured real-scene image frame into the target network model constructed above to obtain the corresponding actual image features; then determine at least one image to be matched for the real-scene image frame according to the set image frame selection rules Frames and corresponding historical image features; finally, according to the similarity value between the actual image certificate and each historical image frame, the closed-loop detection result of the real image frame is determined. The above technical solution, the constructed target network model can quickly and concisely output low-dimensional image feature vectors, and when used for closed-loop detection, on the premise of ensuring the accuracy of closed-loop detection, the required image features in closed-loop detection are effectively reduced. The dimension of the vector shortens the calculation time of the similarity calculation in the closed-loop detection, thereby better meeting the real-time requirements in the closed-loop detection.

Description of drawings

Fig. 1a is a schematic flowchart of a method for constructing a network model provided by Embodiment 1 of the present invention;

Figure 1b shows the topology structure diagram of Convx_1 in the network model constructed in Embodiment 1 of the present invention;

Fig. 1c shows the topology structure diagram of Convx_2 in the network model constructed in Embodiment 1 of the present invention;

Figure 1d shows a schematic diagram of the calculation principle of the C.ReLU calculation function provided by Embodiment 1 of the present invention;

FIG. 1e shows a schematic topology diagram of a training target network model according to an embodiment of the present invention;

Figures 1f to 1m respectively show the visualization diagrams of the output results of each layer in the target network model constructed in Embodiment 1 of the present invention;

FIG. 2 is a schematic flowchart of a closed-loop detection method provided in Embodiment 2 of the present invention;

FIG. 3 is a schematic flowchart of a closed-loop detection method provided by Embodiment 3 of the present invention;

FIG. 4a is a structural block diagram of a device for constructing a network model provided in Embodiment 4 of the present invention;

FIG. 4b is a schematic diagram of a hardware structure of a computer device provided by Embodiment 4 of the present invention;

FIG. 5a is a structural block diagram of a closed-loop detection device provided in Embodiment 5 of the present invention;

FIG. 5b is a schematic diagram of a hardware structure of a computer device provided by Embodiment 5 of the present invention.

detailed description

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, only some structures related to the present invention are shown in the drawings but not all structures.

Embodiment one

Fig. 1a is a schematic flowchart of a method for constructing a network model provided by Embodiment 1 of the present invention, the method is suitable for constructing and training a new network model, and the method can be executed by a device for constructing a network model, wherein, The device can be implemented by software and/or hardware, and is generally integrated into computer equipment.

As shown in Figure 1a, a method for constructing a network model provided by Embodiment 1 of the present invention includes the following operations:

S101. Construct and form an initial network model based on the acquired topology information and configuration parameter information, wherein the topology information includes at least one of the following: the number of convolutional layers, the number of pooling layers, and the number of fully connected layers The number of layers, and the topological connection order between each layer; the configuration parameter information includes at least one of the following: the convolution step size and the convolution kernel size and quantity of each convolution layer, and the pooling step size of each pooling layer And pooling window size, and the number of neurons in each fully connected layer.

In this embodiment, according to the provided topology information, a network model framework can be determined, and according to the provided configuration parameter information, the determined network model framework can be constructed to form an initial network model that can perform image feature extraction calculations. Since the topology information and configuration parameter information are all preset, it can be considered that the number of network layers and the connection order and connection relationship between the layers of the initial network model formed according to the construction are the same as those of the existing neural network. The models are different.

For the network model to be constructed, in addition to the input layer and output layer, its topological structure also includes a convolutional layer, a pooling layer, and a fully connected layer. Specifically, the topology information preset in this embodiment is specifically given to The number of layers of each layer and the overall connection relationship of each layer are shown. For example, in the topology information, the convolutional layer is connected to the input layer, the pooling layer is connected to the convolutional layer, and the fully connected layer is connected to the pooling layer or convolutional layer. In short, based on the above topology information, a network model framework can be constructed.

After forming a network model framework in this embodiment, it is necessary to provide substantial topological connections for each layer in the network model framework based on configuration parameter information, thereby forming an initial network model capable of image feature calculation. The preset configuration parameter information in this embodiment specifically includes the configuration parameters of each layer in the network model to be constructed, and each layer can realize substantial topological connection between adjacent layers based on corresponding configuration parameters.

Exemplarily, the convolutional layer in the network model can establish a convolutional connection capable of performing convolution calculations with the upper layer (maybe the input layer or the pooling layer) through corresponding configuration parameters; the pooling layer can pass The corresponding configuration parameters establish a pooling connection with the upper layer (generally a convolutional layer) that can perform pooling calculations; Full connection for full connection computation.

Specifically, the configuration parameters of the convolution layer include: the size and number of convolution kernels, and the convolution step size. The size of the convolution kernel can be specifically understood as the convolution calculation after the convolution layer establishes a convolution connection with the previous layer. The size of the convolution matrix used at the time; the number of convolution kernels can be specifically understood as the number of different convolution matrices that can be used when performing convolution calculations; the convolution step size can be specifically understood as The movement range when the kernel moves from the current calculation position to the next calculation position, for example, when the convolution step size is 1, it is equivalent to the movement range from the current calculation position to the next calculation position is 1 each time.

Similarly, the configuration parameters of the pooling layer include: the pooling window size and the pooling step size, and the pooling window can be specifically understood as the pooling layer used when performing pooling calculations after establishing a pooling connection with the upper layer The size of the pooling matrix; the pooling step size can be specifically understood as the movement range when the pooling window moves from the current calculation position to the next calculation position in the pooling calculation. In addition, the configuration parameters of the fully connected layer include the number of neurons, and the number of neurons can be specifically used to determine the total number of fully connected weight data required after establishing a fully connected layer with the previous layer.

Further, the convolutional layer includes 5 convolutional layers, which are respectively the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer; the pool The pooling layer includes 2 layers, respectively the first pooling layer and the 2nd pooling layer; the fully connected layer includes 2 layers, respectively the 1st fully connected layer and the 2nd fully connected layer; the topological connection order Expressed as: input layer - the first convolutional layer - the first pooling layer - the second convolutional layer - the second pooling layer - the third convolutional layer - the fourth convolutional layer - the first 5 convolutional layer--the first fully connected layer--the second fully connected layer--the output layer.

In this embodiment, it is preferable to set the hidden layer in the framework of the entire network model to be constructed to be composed of five large convolutional layers, two pooling layers, and two fully connected layers, and each layer has a corresponding name as shown in Section 1. 1 convolutional layer, the first pooling layer, and the first fully connected layer, etc.; this embodiment also provides a preferred topological connection sequence, based on the topological connection sequence, an optimal initial network model framework can be formed.

Further, the i-th convolutional layer includes the i_1th convolutional layer and the i_2th convolutional layer, wherein the value of i is 3, 4 and 5, and the i_1th convolutional layer and the i_2th convolutional layer The convolution calculations of all adopt shortcut connections; the i_jth convolutional layer also includes: the i_j_1th sub-convolutional layer, the i_j_2th sub-convolutional layer and the i_j_3th sub-convolutional layer, wherein the value of j is 1 and 2 .

It can be understood that this implementation further refines the topological structures of the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer, specifically, the third convolutional layer, the fourth convolutional layer layer and the fifth convolutional layer respectively include two small convolutional layers, and the two small convolutional layers included respectively include three sub-convolutional layers. The design can effectively reduce the dimensionality of the image features extracted by the network model. In addition, the two small convolutional layers included use a shortcut connection when performing convolution calculations. The purpose of using the shortcut connection is mainly to speed up the training of the network model when updating and training the weight data of the constructed network model. Convergence time.

In terms of the above optimization techniques, this embodiment preferably sets specific configuration parameters for each layer in the network model. Further, the convolution step of the first convolution layer is preferably 1, the convolution kernel size is preferably 5*5 and the number of convolution kernels is preferably 32; the convolution step of the second convolution layer is preferably It is 1, the size of the convolution kernel is 3*3 and the number of convolution kernels is preferably 64; the pooling steps of the first pooling layer and the second pooling layer are preferably 2 and the pooling window size is preferably 3*3; the number of neurons in the first fully connected layer and the second fully connected layer is preferably 512.

In this embodiment, the first convolutional layer, the second convolutional layer, the first pooling layer, the second pooling layer, the first fully connected layer, and the second fully connected layer correspond to each other in the network model to be constructed. configuration parameters. It can be understood that, since the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer are respectively composed of multiple sub-convolutional layers, this embodiment specifically sets corresponding sub-convolutional layers for each sub-convolutional layer Configuration parameters, and this embodiment gives the preferred configuration parameters of each sub-convolutional layer.

Specifically, on the basis of the above optimization, the convolution step size of the 3_1_1 sub-convolution layer and the 3_2_1 sub-convolution layer are preferably 1, the convolution kernel size is 1*1, and the number of convolution kernels is equal to 1. It is preferably 96; the convolution step size of the 3_1_2 sub-convolution layer and the 3_2_2 sub-convolution layer is preferably 1, the convolution kernel size is 3*3 and the number of convolution kernels is preferably 96; The convolution steps of the 3_1_3th sub-convolution layer and the 3_2_3th sub-convolution layer are preferably 2 and 1 respectively, the size of the convolution kernel is 1*1, and the number of convolution kernels is preferably 192.

Further, the convolution step size of the 4_1_1 sub-convolution layer and the 4_2_1 sub-convolution layer is preferably 1, the convolution kernel size is 1*1, and the number of convolution kernels is preferably 128; the first The convolution steps of the 4_1_2 sub-convolution layer and the 4_2_2 sub-convolution layer are preferably 1, the convolution kernel size is 3*3 and the number of convolution kernels is preferably 128; the 4_1_3 sub-convolution The convolution steps of the layer and the 4_2_3th sub-convolution layer are preferably 2 and 1 respectively, the size of the convolution kernel is 1*1, and the number of convolution kernels is preferably 384.

Further, the convolution step size of the 5_1_1 sub-convolution layer and the 5_2_1 sub-convolution layer is preferably 1, the convolution kernel size is 1*1, and the number of convolution kernels is preferably 256; the first The convolution steps of the 5_1_2 sub-convolution layer and the 5_2_2 sub-convolution layer are preferably 1, the convolution kernel size is 3*3 and the number of convolution kernels is preferably 256; the 5_1_3 sub-convolution The convolution steps of the layer and the 5_2_3th sub-convolution layer are preferably 2 and 1, respectively, the size of the convolution kernel is 1*1, and the number of convolution kernels is preferably 512.

This embodiment provides a parameter information table for network model construction in combination with the above-mentioned optimal setting of configuration parameters of each layer in the network model to be constructed. Layer-corresponding configuration parameters.

Table 1 List of parameter information of the network model to be constructed

As shown in Table 1, the first column in the table indicates the layer identification of each layer in the network model to be constructed, and at the same time implicitly gives the topological connection sequence between layers. In the first column, Convx represents the network The xth convolutional layer in the model, it can be found that when x in Convx takes 3, 4 and 5 respectively, it includes two small convolutional layers, Convx_1 and Convx_2; in the second column, it is specifically given After the network model framework is formed based on each layer in the first column, the type of calculation that each layer has when it is connected with the substantial topology of the adjacent previous layer, for example, the calculation that the convolutional layer has when it is connected with the substantial topology of the adjacent previous layer The type is convolution calculation; in the third column, the fourth column and the fifth column, the optimal configuration parameters corresponding to the convolutional layer and the pooling layer are given in detail, and the above-mentioned filter size is specifically equivalent to the convolutional layer The size of the convolution kernel and the size of the pooling window of the pooling layer, the number of filters mentioned above is specifically equivalent to the number of convolution kernels of the convolution layer; the above step is specifically equivalent to the convolution step of the convolution layer and the pooling The pooling step size of the layer; in the sixth column, the dimension value of the corresponding output result after calculation of each layer in the network model constructed based on the information in the previous columns is specifically given, among which, 365 in the output layer is equivalent to the output The layer has 365 neurons for result output. At the same time, it should be noted that for the fully connected layer, the dimension value of the corresponding output result after the fully connected calculation is actually equal to the number of neurons set in its configuration parameters, which is preferably set to 512 in this embodiment.

It can be understood that when x in the above-mentioned Convx is 3, 4 and 5 respectively, the corresponding two small convolutional layers respectively contain 3 sub-convolutional layers. In this embodiment, each sub-convolutional layer is also The corresponding configuration parameters are set. FIG. 1 b shows the topology structure diagram of Convx_1 in the network model constructed in Embodiment 1 of the present invention; FIG. 1 c shows the topology structure diagram of Convx_2 in the network model constructed in Embodiment 1 of the present invention. As shown in Figure 1b and Figure 1c, the default values of x in this embodiment are 3, 4, and 5 respectively. The main topology connections in the topology diagrams of Convx_1 and Convx_2 are specifically composed of 3 convolutional layers, and the convolution kernel sizes are 1 ×1, 3×3 and 1×1, where the 1×1 convolution kernel is used to control the feature dimension of the image feature extraction, which can reduce the input and output dimensions of the 3×3 convolution kernel.

At the same time, it can also be seen from Fig. 1b and Fig. 1c that both Convx_1 and Convx_2 use a shortcut connection 110, that is, when Convx_1 and Convx_2 perform convolution calculations, the feature data input to Convx_1 and Convx_2 respectively can first be based on the setting Carry out convolution calculation with the given topology structure to obtain the output feature data, and then add the obtained output feature data to the input feature data again, and the result of the sum calculation can be used as the output result of Convx_1 and Convx_2. It should be noted that the shortcut connection 110 in Convx_1 is additionally connected topologically with a convolutional layer of a 1×1 convolutional kernel, and the above-mentioned additional convolutional layer can be specifically used to ensure that the two sets of feature data participating in the sum calculation Have the same dimension, so as to ensure that the sum calculation can be performed normally, and because the input feature data of Convx_2 is the output feature data of Convx_1, because the input and output dimensions of Convx_1 and Convx_2 are the same, there is no need to add 1×1 convolution to Convx_2 The kernel's convolutional layer.

In addition, it can also be found that each convolutional layer in the topological connections of Convx_1 and Convx_2 is followed by batch normalization (Batch Normalization, BN). The purpose of BN operation is to speed up the training convergence speed of the constructed network model. It can be understood that, in this embodiment, the Relu activation function is also used after the BN operation to enhance the expression strength of the output results of each convolutional layer.

It should be noted that, except that the BN operation is followed after each convolutional layer given in the above-mentioned Fig. 1b and Fig. 1c, it is preferred in this embodiment that the BN operation is set after each convolutional layer in the constructed network model, such as The BN operation is also set after the first convolutional layer and the second convolutional layer, and the BN operation is specifically completed before entering the pooling layer. The purpose of setting the BN operation is also to speed up the convergence speed of the constructed network model.

S102. According to the acquired training and learning information, iteratively train the initial network model to obtain a target network model with a standard weight data set.

In this embodiment, the above steps may construct an initial network model capable of extracting image features according to preset topology information and configuration parameter information. It is understandable that in the initial network model constructed, only the types of calculations that can be performed after the substantial topological connections of each layer are given, and the weight data required for substantial calculations are not given. The weight data can refer to volume The specific value in the convolution kernel used in the convolution calculation of the product layer can also be the specific value in the pooling window used in the pooling calculation of the pooling layer, or it can be compared with each neuron in the fully connected layer. The connection weight value at the time of connection, therefore, based on the initial network model, the correct image feature result cannot be output directly.

The purpose of this step is to provide initial weight data for each layer in the initial network model, and through corresponding iterative update steps to update the corresponding weight data of each layer, and finally obtain the goal that each layer has the best weight data Network model to achieve accurate extraction of image features based on the target network model. This step specifically performs training of the network model based on preset training and learning information.

Specifically, the training and learning information includes at least one of the following: input image sample set, activation function, bias data, initial weight data and convolution function of the convolution kernel in each convolution layer, pooling in each pooling layer The initial weight data and the pooling function of the window, the initial weight data and the output classification function of the neurons in each fully connected layer; the standard weight data set includes at least one of the following: in each convolution layer after iterative training Convolution kernel, pooling windows in each pooling layer, and standard weight data corresponding to neurons in each fully connected layer.

In this embodiment, the input image sample set is Places365-Standard, a large data set in the field of scene recognition. The Places365-Standard data set contains more than 1.8 million scene pictures with corresponding scene identifications. There are 365 scene pictures in total. The scene category, that is, the scene identifier of each scene picture belongs to one of the 365 scene categories. The activation function can specifically be used to increase non-linear factors, thereby enhancing the expressive ability of the network model, and the activation function in this embodiment preferably adopts the ReLu activation function. The bias data is preferably set to 0.

In addition, the training and learning information also includes the initial weight data and calculation functions required for the calculation of each layer. In this embodiment, the Xavier initial algorithm is preferably used to set the initial weight data for each layer in the network model. At the same time, this In the embodiment, the convolution function of the first convolutional layer and the second convolutional layer is set as the C.ReLU calculation function; in this embodiment, the pooling function of the first pooling layer and the second pooling layer is set to the maximum Pooling calculation function to reduce the feature dimension of the output data of the connected convolutional layer.

For the C.ReLU calculation function, the calculation principle is: perform convolution calculation based on the set convolution kernel to obtain the current actual eigenvalue, then invert each actual eigenvalue to obtain the inverse eigenvalue, and then calculate the actual eigenvalue Concatenate with the anti-eigenvalue, and then use the ReLu activation function to adjust the nonlinear factors. The dimension value of the finally obtained feature data is equivalent to twice the number of convolution kernels set. Figure 1d shows a schematic diagram of the calculation principle of the C.ReLU calculation function provided by Embodiment 1 of the present invention. Figure 1d describes the calculation process of the C.ReLU calculation function in the form of a diagram, that is, the entire calculation process is convolution first, Afterwards, the convolution result is reversed, and then the reversed result is concatenated with the convolution result, and finally processed through the ReLU activation function, and the feature data after the dimension value is doubled is output.

Exemplarily, this embodiment sets that both the first convolutional layer and the second convolutional layer use the C.ReLU calculation function for convolution calculation, as shown in Table 1, the number of convolution kernels of the first convolutional layer is 32, the dimension value of the corresponding output feature data is 128*128*64, it can be understood that 128*128 is the dimension value of the input of the previous layer, and the corresponding dimension value after ordinary convolution calculation is 128*128*32, but the dimension value of the output feature data after convolution calculation based on the C.ReLU calculation function is twice as large as before. Based on this convolution calculation method, the calculation amount of convolution calculation in the constructed network model can be reduced, and the calculation time can be saved under the premise of ensuring the accuracy of the calculation results.

In addition, in this embodiment, the output layer is determined as a classification function, which is used to realize the classification of image scenes through the calculated image features, thereby adjusting the corresponding weights of each layer in the constructed initial network model through the output scene classification results data (which may be the initial weight data, or the adjusted weight data to be adjusted), and then obtain a new scene classification result again according to the adjusted weight data, and so on until the loop end condition is reached.

The output classification in this embodiment adopts a Sofmax classifier, and its corresponding output classification result is the probability p(y=j|x) that each sample image x belongs to each scene category j. Specifically, for the i-th sample image x ⁽ⁱ⁾ , the corresponding classification function h(x ⁽ⁱ⁾ ) can be expressed as:

Among them, θ represents the weight data parameter matrix composed of the weight data of each layer in the initial network model constructed; k is the number of categories; y is the category label vector.

Exemplarily, before inputting the sample image x ⁽ⁱ⁾ into the initial network model, this embodiment preferably performs grayscale, mean subtraction and padding preprocessing on it and reduces it to 128*128, the processed sample image After inputting the initial network model, a k-dimensional probability vector p will eventually be obtained, and then through the formula

class of scenes for which x ⁽ⁱ⁾ can be predicted

In addition, in order to update and iteratively train the initial network model in this embodiment, a weight data parameter matrix θ is formed based on the current weight data of each layer in the initial network model, and a loss function L(θ ), L(θ) is used to determine the corresponding loss value when calculating image features based on the current weight data parameter matrix θ, where the loss function L(θ) is expressed as:

Among them, m is the number of sample images in the selected sample image set; θ represents the weight data parameter matrix; x ⁽ⁱ⁾ represents the i-th sample image in the sample image set; y ⁽ⁱ⁾ represents the actual value of the i-th sample image Scene category; k represents the number of scene categories.

In this embodiment, after determining the training and learning information required for training, the implementation steps of training the initial network model as the target network model are specifically given:

1) Randomly select a set number of sample images from more than 1.8 million scene images as the sample image set;

2) Input the selected sample image set into the initial network model, and output the current corresponding actual scene category of each sample image through the output layer through the above formula (1) and formula (2);

3) According to the above formula (3) and the actual scene category corresponding to the current t-1 iteration of each sample image, determine the loss value corresponding to the current weight data parameter matrix θ _t-1 ;

4) Update the current weight data parameter matrix θ _t-1 through stochastic gradient descent of the impulse to be expected.

Specifically, by V _t =λ·V _t-1 -η·▽L(θ _t-1 ) (4)

and θ _t = θ _t-1 +V _t (5)

Realize the update of the weight data parameter matrix θ _t-1 , wherein, t represents the tth iteration of the weight data parameter matrix; λ is the impulse coefficient, preferably 0.9, V _t is the tth time of the weight data parameter matrix The corresponding update value during iteration, θt is the weight data parameter matrix at the tth iteration, η is the learning rate, the initial preference is 0.01, ▽L(θ _t-1 ) means L(θ _t-1 ) vs θ _t Derivative value of _-1 .

Based on the above steps, before the iteration converges, the weight data parameter matrix of the initial neural network model can be iteratively updated to obtain the weight data parameter matrix required for the next iteration. In this way, the final standard weight data parameter matrix can be obtained during iteration convergence, and the target network model after training and learning is formed according to the standard weight data parameter matrix.

It should be noted that all the values in the standard weight data parameter matrix constitute the standard weight data set, and the standard weight data set specifically includes the convolution kernel in each convolution layer after iterative training, and the data in each pooling layer. Pooling window and standard weight data corresponding to neurons in each fully connected layer. That is to say, the initial network model forms a target network that can accurately extract image features based on the standard weight data corresponding to the convolution kernel in each convolutional layer, the pooling window in each pooling layer, and the neurons in each fully connected layer. Model.

When the target network model trained in this embodiment performs image scene recognition on the scene images in the Places365-Standard data set, the accuracy rate of one actual scene category obtained is directly equivalent to the scene identification of the scene image can reach 50.16%. , and the accuracy rate of the scene identification of the scene image in the obtained five candidate scene categories can reach 80.03%.

Figure 1e shows a schematic topology diagram of a training target network model according to an embodiment of the present invention. As shown in Figure 1e, the entire topology generally includes the input layer-the first convolutional layer-the first pooling layer-the first 2 convolutional layers--2nd pooling layer--3rd convolutional layer--4th convolutional layer--5th convolutional layer--1st fully connected layer--2nd fully connected layer--output layer , the input layer input is a 128*128 single-channel image, the first convolutional layer and the second convolutional layer use the C.ReLU calculation function to perform convolution calculations on the input data, and the activation function used in the calculation Both are ReLu activation functions. After the convolution calculation of the fifth convolutional layer, the image feature vector with a dimension value of 4*4*512 is output, and then the 4*4*512 The image features are fused and classified, so that the image feature vector with a dimension value of 512 dimensions is output, and the subsequent second fully connected layer performs full connection calculation on the image feature vector with a dimension value of 512 dimensions, and the output dimension value is also 512 dimensions. The image feature vector of the image finally passes through the Sofmax classifier of the output layer, and outputs the probability value of the input image relative to 365 scene categories through 365 neurons of the Sofmax classifier.

In this implementation, for the target network model diagram formed by training, the first and second convolutional layers are specifically used to extract features such as image edge gradients and color blocks of the input image, and the third to fifth convolutional layers It is specifically used to extract the image local semantic features of the input image and gradually extract the global semantic features of the input image. Figures 1f to 1m respectively show the visualization diagrams of the output results of each layer in the target network model constructed in Embodiment 1 of the present invention.

Specifically, Figure 1f shows the preprocessed input image, and it can be seen that the image features in the image are very clear; Figure 1g shows the output after the convolution calculation of the first convolutional layer As a result, it can be seen that the image clearly shows the outline of the image; Figure 1h specifically shows the output results after the pooling calculation of the first pooling layer, and the outline of the image can still be vaguely judged in this figure; while the subsequent figure From 1i to 1m, as the number of convolutional layers in the target network model deepens, the receptive field of neurons gradually increases, and the extracted image features are gradually abstracted, making it difficult for the human eye to distinguish the features of the image. But for image processing, the more abstract the extracted image features are, the stronger the representation ability it has, and the subsequent image processing can be performed more accurately based on the image features with strong representation ability.

The target network model formed by training in this embodiment has the following specific characteristics: 1) In view of the real-time requirements of image processing, the convolution calculation of the first convolutional layer and the second convolutional layer in the target network model uses C.ReLU calculation function, and a large number of convolutional layers with 1*1 convolution kernels are used, which greatly reduces the calculation amount of the target network model and speeds up the calculation speed; 2) for the difficult training and convergence of convolutional neural networks , the BN operation is used after all convolutional layers in the target network model, and shortcut connections are used in Convx (x=3,4,5), thereby speeding up the convergence speed of the target network model training.

Although the training of the constructed initial network model takes a long time, the speed of image feature extraction using the trained target network model is very fast. After testing on the GPU, the target network model of this embodiment is used for image feature extraction. The time is about 0.0098s. In addition, compared with the existing PlaceCNN convolutional network model, the total amount of weight data in the target network model of this embodiment is only one-seventh of the PlaceCNN convolutional network model, so it can be considered that this implementation The network topology of the target network model is better than the PlaceCNN convolutional network model, and it is more suitable for image feature extraction in image processing.

The embodiment of the present invention provides a method for constructing a network model. The constructed target network model can output low-dimensional image feature vectors quickly and concisely, and the extraction result is not affected by the image feature extraction based on the constructed target network model. At the same time, the extracted image features can ensure the accuracy of the processing results when the image processing is performed, thus ensuring the processing effect of the image processing.

Embodiment two

Fig. 2 is a schematic flow chart of a closed-loop detection method provided by Embodiment 2 of the present invention. The method is suitable for performing closed-loop detection in real-time positioning and map construction. The method can be executed by a closed-loop detection device, which can be controlled by software. And/or hardware implementation, and generally integrated in computer equipment capable of real-time positioning and map construction.

As shown in Figure 2, a closed-loop detection method provided by Embodiment 2 of the present invention specifically includes the following operations:

S201. Input the currently captured real-scene image frame into a preset target network model to obtain actual image features of the real-scene image frame.

In this embodiment, real-time positioning and closed-loop detection during map construction are specifically realized. In this step, firstly, the captured real-scene image frames are input into the target network model. It can be known that the target network model is determined based on the method for constructing the network model provided by the above-mentioned embodiments of the present invention. Thereby, the actual image features of the real-scene image frame can be obtained.

It can be understood that the output layer in the target network model can be set differently according to the actual problem to be dealt with. Exemplarily, when performing loop closure detection in this embodiment, the scene classification result corresponding to the input image is not required, but the image frame feature vector of the input image is required, because this embodiment can preferably use the first full connection in the target network model Layer or the second fully connected layer calculates the feature vector obtained as the result output, thus, for each input image, a 512-dimensional image feature vector can be obtained. From this, it can be found that the actual image features of the real-scene image frame have a relatively low dimensional value, which is convenient for loop closure detection in real time.

At the same time, it can be understood that in this step, before the real-scene image frame is input into the target network model as an input image, operations such as grayscale, mean value subtraction, and pixel resolution adjustment are first performed on the real-scene image frame, thereby forming a 128*128 single Channel's real-world image frame.

S202. Determine at least one image frame to be matched of the real-scene image frame according to the set image frame selection rule, and acquire historical image features of each image frame to be matched.

Specifically, in order to determine the closed-loop detection result of the real-scene image frame, it is necessary to select an image frame to be matched for similarity matching for the real-scene image frame. It should be noted that the image data processed by closed-loop detection often has time continuity. In general, the feature correlation between the above-mentioned real-scene image frame and its adjacent image frames is relatively large, and the calculated similarity value is often very high. Therefore, Adjacent image frames are easily misdetected as closed-loop regions of real-scene image frames.

In this embodiment, a selection rule for selecting image frames to be matched may be set, so as to avoid similarity matching between real-scene image frames and adjacent image frames. Specifically, the image frames to be matched can be selected from captured historical image frames based on image frame selection rules, and at the same time, historical image features corresponding to each image frame to be matched can also be obtained. It can be understood that each image to be matched The historical image features of the frames are also extracted through the target network model. In this embodiment, the captured real-scene image frames and the determined actual image features can be stored in the set location, thereby forming a historical image frame and corresponding historical image features. database.

In this embodiment, the number of interval frames between the selected image frame to be matched and the real-scene image frame may be set in the image frame selection rule.

S203. Based on the similarity value between the actual image feature and each historical image feature, determine a closed-loop detection result of the real-scene image frame.

This embodiment can determine the similarity value between the actual image feature and each historical image feature based on the similarity calculation formula of the feature vector (such as the cosine value of the two feature vectors), and can compare each similarity value with the set threshold at the same time, Then determine the closed-loop detection result of the real-scene image frame according to the comparison result.

A closed-loop detection method provided by Embodiment 2 of the present invention can extract image features from captured image frames based on the target network model constructed in the above-mentioned embodiment, and effectively reduce the closed-loop detection accuracy under the premise of ensuring the accuracy of closed-loop detection. The dimensions of the required image feature vectors in the image, thereby shortening the calculation time of the similarity calculation in the closed-loop detection, thus better meeting the real-time requirements of the closed-loop detection.

Embodiment three

Fig. 3 is a schematic flow chart of a closed-loop detection method provided by Embodiment 3 of the present invention. The embodiment of the present invention is optimized on the basis of the above-mentioned embodiments. In this embodiment, further according to the set image frame selection rules, determine At least one image frame to be matched of the real-scene image frame, and the historical image features of each image feature to be matched are obtained, which is embodied as: obtaining the set interval frame number and the frame number of the real-scene image frame, and the frame number The difference with the number of frames at intervals is determined as the target frame number; in the built historical information base, the historical image frame whose frame number is less than or equal to the target frame number is used as the image frame to be matched; each image frame to be matched is obtained Historical image features determined based on the target network model.

Further, this embodiment will also determine the closed-loop detection result of the real-scene image frame based on the similarity value between the actual image feature and each historical image feature, and the specific optimization is: calculate the actual image feature and each historical image feature The similarity value; the image frame to be matched corresponding to the similarity value greater than the set similarity threshold is determined as a candidate closed-loop image frame, and the candidate closed-loop image frame is added to the set candidate closed-loop set; if the candidate closed-loop If there is only one candidate closed-loop image frame in the set, the candidate closed-loop image frame is determined as the closed-loop area of the real-scene image frame; if there are at least two candidate closed-loop image frames in the candidate closed-loop set, then based on the set closed-loop determination The strategy is to obtain the closed-loop area of the real-scene image frame.

As shown in Figure 3, a closed-loop detection method provided by Embodiment 3 of the present invention specifically includes the following operations:

S301. Input the currently captured real-scene image frame into a preset target network model to obtain actual image features of the real-scene image frame.

It should be noted that the device for real-time positioning and map building can capture images based on the camera on it. Generally, the camera can continuously capture images, and the speed of image capture is much faster than that of the device for real-time positioning and map building. The moving speed of the camera causes the multiple image frames continuously captured by the camera to actually be images of the same scene.

If the loop closure detection is performed for each image frame captured by the camera in this embodiment, the processing and calculation burden of the device will be increased to a certain extent. Therefore, this embodiment considers adjusting the image capture frequency of the camera, and preferably adjusts the capture frequency to be the same as the moving rate of the device.

In this step, it can be considered that the camera captures the real-scene image frame based on the same capture frequency as the device's moving rate, and then obtains the actual image features of the real-scene image frame. The following S302-S304 of this embodiment specifically provides the selection operation of the image frame to be matched.

S302. Obtain the set interval frame number and the frame number of the real-scene image frame, and determine the difference between the frame number and the interval frame number as the target frame number.

Specifically, the number of interval frames can be specifically understood as the minimum interval between the selected image frame to be matched and the real-scene image frame, and the interval frame number can be preset in the image frame selection rule, and the interval The number of frames can be actually set based on the actual image environment, and the preferred value range of the setting can be [300,800].

The frame number of the real-scene image frame is specifically formed when it is captured, and can be used as an ID identification to distinguish it from other image frames. Generally, only when the frame number of the real-scene image frame is greater than the number of interval frames, the closed-loop detection of the real-scene image frame is performed, otherwise, the closed-loop detection of the real-scene image frame is directly ignored, and the next frame of image is directly performed. capture.

In this step, the difference between the frame number of the real-scene image frame and the number of interval frames is determined as the target frame number, and the target frame number can be specifically understood as the maximum frame number that the selected image frame to be matched can have.

S303. Using the historical image frame whose frame number is less than or equal to the target frame number in the constructed historical information base as the image frame to be matched.

In this embodiment, the captured historical image frames and their corresponding historical image features can be stored in the set historical information database. In this embodiment, the currently captured real-scene image frames and corresponding actual image features can be added in real time To the historical information base, realize the dynamic update of the historical information base.

In this step, all historical image frames whose frame numbers are less than or equal to the target frame number can be used as image frames to be matched, or key image frames can be selected from qualified historical image frames as image frames to be matched. Specifically, key image frames may be selected from historical image frames less than or equal to the target frame number, and the difference value may be equal to 1% of the number of interval frames.

S304. Acquire historical image features determined based on the target network model for each image frame to be matched.

S305. Calculate the similarity value between the actual image feature and each historical image feature.

Exemplarily, the similarity value is calculated based on the feature vector similarity calculation formula.

S306. Determine the image frame to be matched corresponding to the similarity value greater than the set similarity threshold as a candidate closed-loop image frame, and add the candidate closed-loop image frame to the set candidate closed-loop set.

In this step, each similarity value obtained through calculation may be compared with a set similarity threshold, and the set similarity threshold may preferably be 0.9. When there is a similarity value greater than the set similarity threshold, the image frame to be matched corresponding to the similarity value may be used as a candidate closed-loop image frame and added to the candidate closed-loop set.

In this embodiment, the operation of this step can be performed on all the image frames to be matched corresponding to the similarities meeting the similarity determination conditions. In this embodiment, subsequent statistics may be performed on the number of image frames included in the candidate closed-loop set, and it may be determined whether to perform S307 or S308 according to the statistical results.

S307. If there is only one candidate closed-loop image frame in the candidate closed-loop set, determine the candidate closed-loop image frame as a closed-loop area of the real-scene image frame.

Specifically, in this step, when there is only one candidate closed-loop image frame in the candidate closed-loop set, the candidate closed-loop image frame can be directly determined as the closed-loop area of the real-scene image frame, that is, the scene in the real-scene image frame is considered to be the same as the candidate closed-loop The scene of the image frame is the same area.

S308. If there are at least two candidate closed-loop image frames in the candidate closed-loop set, obtain a closed-loop area of the real-scene image frame based on a set closed-loop determination strategy.

Specifically, when there are multiple candidate closed-loop image frames in the candidate closed-loop set, each candidate closed-loop image frame cannot be directly determined as the closed-loop area of the real-scene image frame, and it is necessary to determine whether the closed-loop condition is satisfied based on the closed-loop determination strategy.

Further, the closed-loop determination strategy based on the setting to obtain the closed-loop area of the real-scene image frame includes: when the frame numbers of the candidate closed-loop image frames in the candidate closed-loop set are all discrete, determining the real-scene image frame There is no closed-loop area; when there are candidate closed-loop image frames with continuous frame numbers in the candidate closed-loop set, determine the start frame number and the end frame number with continuous frame numbers, and based on the start frame number to the end frame number The corresponding candidate closed-loop image frames form a historical image area, and the historical image area is determined as the closed-loop area of the real-scene image frame.

Specifically, the frame numbers of each candidate closed-loop image frame are acquired first, and it is determined whether the frame numbers are discrete or continuous. It should be noted that if this embodiment selects the image frame to be matched from the historical image frames based on the difference value, then this step needs to determine whether the frame number difference of the adjacent candidate closed-loop image frame is equal to the set difference value, If they are equal, it can also be considered that the frame numbers of adjacent candidate closed-loop image frames are consecutive.

This step can determine that there is no closed-loop area in the real-scene image frame when the frame number is discrete; it can also be used to synthesize the historical image area based on candidate closed-loop image frames corresponding to all continuous frame numbers when the frame number is continuous, and determine the historical image area as the The closed-loop region of the reality image frame.

It can be understood that there may be multiple consecutive frame number segments in the candidate closed-loop set. In this embodiment, it can be considered that the historical image areas corresponding to the multiple frame number consecutive segments are all closed-loop areas of the real-scene image frame, because the multiple frames The historical image area corresponding to the continuous segment of the number may be the same area that the device passed through in different time periods.

A closed-loop detection method provided by Embodiment 3 of the present invention specifically describes the selection process of the actual image frame to be matched, and also describes the operation process of determining the closed-loop area of the actual image frame among the candidate closed-loop image frames. Using this method, the target network model is firstly used to obtain the low-dimensional image features of the real-scene image frame and the image frame to be matched, thereby better reducing the closed-loop detection while ensuring the accuracy of the similarity calculation results of the closed-loop detection. The calculation time of the similarity degree in the detection can better meet the real-time requirement in the closed-loop detection.

Embodiment four

Figure 4a is a structural block diagram of a device for constructing a network model provided by Embodiment 4 of the present invention, which is suitable for constructing and training a new network model, which can be implemented by software and/or hardware, and generally integrated in computer equipment. As shown in FIG. 4 a , the device includes: an initial construction module 41 and a target determination module 42 .

Wherein, the initial construction module 41 is configured to construct and form an initial network model based on the acquired topology information and configuration parameter information, wherein the topology information includes at least one of the following: the number of convolutional layers, the number of pooling layers The number of layers, the number of layers of the fully connected layer, and the topological connection sequence between the layers; the configuration parameter information includes at least one of the following: the convolution step size of each convolution layer and the size and number of convolution kernels, each pool The pooling step size and pooling window size of the layer, and the number of neurons in each fully connected layer.

The target determination module 42 is configured to iteratively train the initial network model according to the acquired training and learning information, and obtain a target network model with a standard weight data set.

In this embodiment, the device first constructs an initial network model based on the acquired topology information and configuration parameter information through the initial construction module 41; then iteratively trains the initial network model based on the target determination module 42 according to the acquired training and learning information model to obtain a target network model with a standard weight dataset.

The network model construction device provided in Embodiment 4 of the present invention can form a specific initial network model according to the specially set topology information and configuration parameter information, and can obtain the target network model through training, while ensuring low-dimensional output image features, and the image feature extraction results based on the target network model constructed by the device are not affected by the lighting environment. The processing effect of image processing.

Further, the convolutional layer includes 5 convolutional layers, which are respectively the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer; the pool The pooling layer includes 2 layers, respectively the first pooling layer and the 2nd pooling layer; the fully connected layer includes 2 layers, respectively the 1st fully connected layer and the 2nd fully connected layer; the topological connection order Expressed as: input layer - the first convolutional layer - the first pooling layer - the second convolutional layer - the second pooling layer - the third convolutional layer - the fourth convolutional layer - the first 5 convolutional layer--the first fully connected layer--the second fully connected layer--the output layer.

Further, the convolution step of the first convolution layer is 1, the convolution kernel size is 5*5, and the number of convolution kernels is 32; the convolution step of the second convolution layer is 1, and the convolution kernel size is 1. The size of the product kernel is 3*3 and the number of convolution kernels is 64; the pooling steps of the first pooling layer and the second pooling layer are both 2 and the pooling window size is 3*3; The number of neurons in the fully connected layer 1 and the fully connected layer 2 is 512.

Further, the i-th convolutional layer includes the i_1th convolutional layer and the i_2th convolutional layer, wherein the value of i is 3, 4 and 5, and the i_1th convolutional layer and the i_2th convolutional layer The convolution calculations of all adopt shortcut connections; the i_jth convolutional layer also includes: the i_j_1th sub-convolutional layer, the i_j_2th sub-convolutional layer and the i_j_3th sub-convolutional layer, wherein the value of j is 1 and 2 .

On the basis of the above-mentioned optimization, the convolution step size of the 3_1_1 sub-convolution layer and the 3_2_1 sub-convolution layer can be preferably 1, the convolution kernel size can be preferably 1*1 and the number of convolution kernels is equal to It can be preferably 96; the convolution step size of the 3_1_2 sub-convolution layer and the 3_2_2 sub-convolution layer can be preferably 1, the convolution kernel size can be preferably 3*3 and the number of convolution kernels It can be preferably 96; the convolution step size of the 3_1_3 sub-convolution layer and the 3_2_3 sub-convolution layer can be preferably 2 and 1 respectively, and the convolution kernel size can be preferably 1*1 and convolution kernel The number can be preferably 192.

Further, the convolution step size of the 4_1_1 sub-convolution layer and the 4_2_1 sub-convolution layer can be preferably 1, the convolution kernel size can be preferably 1*1 and the number of convolution kernels can be preferably 128 The convolution step size of the 4_1_2 sub-convolution layer and the 4_2_2 sub-convolution layer can be preferably 1, the convolution kernel size can be preferably 3*3 and the convolution kernel quantity can be preferably 128 The convolution step size of the 4_1_3 sub-convolution layer and the 4_2_3 sub-convolution layer can be preferably 2 and 1 respectively, the convolution kernel size can be preferably 1*1 and the convolution kernel quantity can be preferably for 384.

Further, the convolution step size of the 5_1_1st sub-convolution layer and the 5_2_1 sub-convolution layer can be preferably 1, the size of the convolution kernel can be preferably 1*1 and the number of convolution kernels can be preferably 256 The convolution step size of the 5_1_2 sub-convolution layer and the 5_2_2 sub-convolution layer can be preferably 1, the convolution kernel size can be preferably 3*3 and the convolution kernel quantity can be preferably 256 The convolution step size of the 5_1_3 sub-convolution layer and the 5_2_3 sub-convolution layer can be preferably 2 and 1 respectively, the convolution kernel size can be preferably 1*1 and the convolution kernel quantity can be preferably for 512.

On the basis of the above optimization, the training and learning information includes at least one of the following: input image sample set, activation function, bias data, initial weight data and convolution function of the convolution kernel in each convolution layer, each pool The initial weight data and the pooling function of the pooling window in the pooling layer, the initial weight data and the output classification function of the neurons in each fully connected layer; the standard weight data set includes at least one of the following: after iterative training, each The standard weight data corresponding to the convolution kernel in the convolutional layer, the pooling window in each pooling layer, and the neurons in each fully connected layer.

At the same time, the embodiment of the present invention also provides a computer device. FIG. 4b is a schematic diagram of the hardware structure of a computer device provided in Embodiment 4 of the present invention. As shown in FIG. 4b, the computer device provided in Embodiment 4 of the present invention includes : Processor 401 and storage device 402, the processor in this computer equipment can be one or more, take a processor 401 as an example in Fig. 4 b, the processor and storage device in the described computer equipment can pass bus or other connection mode, in Figure 4b, the bus connection is taken as an example.

The storage device 402 in the computer equipment, as a computer-readable storage medium, can be used to store one or more programs, and the programs can be software programs, computer executable programs and modules, such as the network model provided by the embodiment of the present invention The corresponding program instructions/modules in the construction device (for example, the modules shown in FIG. 4a, including: the initial construction module 41 and the target determination module 42). The processor 401 executes various functional applications and data processing of the computer equipment by running the software programs, instructions and modules stored in the storage device 402, that is, implements the method for constructing the network model in the above method embodiments.

The storage device 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the device, and the like. In addition, the storage device 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some examples, the storage device 402 may further include memories that are remotely located relative to the processor 401, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Moreover, when one or more programs included in the above-mentioned computer device are executed by the one or more processors 401, one of the programs may perform the following operations:

Based on the obtained topology information and configuration parameter information, construct an initial network model, wherein the topology information includes at least one of the following: the number of layers of the convolutional layer, the number of layers of the pooling layer, and the number of layers of the fully connected layer , and the topological connection order between each layer; the configuration parameter information includes at least one of the following: the convolution step size and the convolution kernel size and quantity of each convolution layer, the pooling step size and the pooling step size of each pooling layer Optimize the window size and the number of neurons in each fully connected layer; according to the obtained training and learning information, iteratively train the initial network model to obtain a target network model with a standard weight data set.

In addition, an embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method for constructing a network model provided in Embodiment 1 of the present invention is implemented, wherein the above-mentioned Embodiment 1 The provided method includes: constructing and forming an initial network model based on the acquired topology information and configuration parameter information, wherein the topology information includes: the number of convolutional layers, the number of pooling layers, and the layers of a fully connected layer number and the topological connection sequence between each layer; the configuration parameter information includes: the convolution step size and the convolution kernel size and quantity of each convolution layer, the pooling step size and the pooling window size of each pooling layer, and the number of neurons in each fully connected layer; according to the obtained training and learning information, iteratively train the initial network model to obtain a target network model with a standard weight data set.

Embodiment five

Figure 5a is a structural block diagram of a closed-loop detection device provided by Embodiment 5 of the present invention, which is suitable for performing closed-loop detection in real-time positioning and map construction, and the device can be realized by software and/or hardware, and It is generally integrated in computer equipment capable of real-time positioning and map construction. As shown in FIG. 5 , the device includes: a feature extraction module 51 , an image selection module 52 and a detection determination module 53 .

Among them, the feature extraction module 51 is used to input the currently captured real-scene image frame into the preset target network model to obtain the actual image features of the real-scene image frame. The target network model is based on the network provided by the fourth embodiment of the present invention. The construction device of the model is determined;

The image selection module 52 is used to determine at least one image frame to be matched and corresponding historical image features according to the set image frame selection rules;

The detection and determination module 53 is configured to determine the closed-loop detection result of the real-scene image frame based on the similarity value between the actual image feature and each historical image feature.

In this embodiment, the device first inputs the currently captured real-scene image frame into the preset target network model through the feature extraction module 51 to obtain the actual image features of the real-scene image frame, and then uses the image selection module 52 according to the set The image frame selection rule determines at least one image frame to be matched and the corresponding historical image feature; finally, the closed-loop detection of the real-scene image frame is determined based on the similarity value between the actual image feature and each historical image feature through the detection determination module 53 result.

Embodiment 5 of the present invention provides a closed-loop detection device, which can extract image features of captured image frames based on the target network model constructed in the above-mentioned embodiments, and effectively reduce the closed-loop detection accuracy under the premise of ensuring the accuracy of closed-loop detection. The dimensions of the required image feature vectors in the image, thereby shortening the calculation time of the similarity calculation in the closed-loop detection, thus better meeting the real-time requirements of the closed-loop detection.

Further, the image selection module 52 is specifically used for:

Obtain the set interval frame number and the frame number of the real-scene image frame, and determine the difference between the frame number and the interval frame number as the target frame number; set the frame number less than or equal to The historical image frame of the target frame number is used as the image frame to be matched; and the historical image features determined based on the target network model of each image frame to be matched are obtained.

Further, the detection determination module 53 includes:

A similar calculation unit, used to calculate the similarity value between the actual image feature and each historical image feature;

A candidate determination unit, configured to determine the image frame to be matched corresponding to the similarity value greater than the set similarity threshold as a candidate closed-loop image frame, and add the candidate closed-loop image frame to the set candidate closed-loop set;

A first determining unit, configured to determine the candidate closed-loop image frame as the closed-loop area of the real-scene image frame when there is only one candidate closed-loop image frame in the candidate closed-loop set;

The second determining unit is configured to obtain the closed-loop area of the real-scene image frame based on a set closed-loop determination strategy when there are at least two candidate closed-loop image frames in the candidate closed-loop set.

On the basis of the above optimization, the second determining unit is specifically used for:

When the frame numbers of the candidate closed-loop image frames in the candidate closed-loop set are all discrete, it is determined that there is no closed-loop area in the real-scene image frame; when there are candidate closed-loop image frames with continuous frame numbers in the candidate closed-loop set, Determine the start frame number and the end frame number with consecutive frame numbers, and form a historical image area based on the corresponding candidate closed-loop image frames between the start frame number and the end frame number, and determine the historical image area as the real-scene image The closed-loop region of the frame.

At the same time, Embodiment 5 of the present invention also provides a computer device. FIG. 5b is a schematic diagram of the hardware structure of a computer device provided by Embodiment 5 of the present invention. As shown in FIG. 5b, the computer device provided by Embodiment 5 of the present invention includes : Camera 501, for capturing image frames, also includes: processor 502 and storage device 503, the processor in this computer equipment can be one or more, take a processor 502 as example in Fig. 5b, described computer equipment The camera in the camera can be respectively connected to the processor and the storage device through a bus or other methods, and the processor and the storage device are also connected through a bus or other methods. In FIG. 5b, connection through a bus is taken as an example. It can be understood that the processor 502 in the computer device can control the operation of the camera 501 .

The storage device 503 in the computer equipment, as a computer-readable storage medium, can be used to store one or more programs, and the programs can be software programs, computer-executable programs and modules, such as the closed-loop detection provided by the embodiment of the present invention The corresponding program instructions/modules in the device (for example, the modules shown in Fig. 5a, including: feature extraction module 51, image selection module 52 and detection determination module 53). The processor 502 executes various functional applications and data processing of the computer equipment by running the software programs, instructions and modules stored in the storage device 503 , that is, realizes the closed-loop detection method in the above method embodiment.

The storage device 503 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the device, and the like. In addition, the storage device 503 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some examples, the storage device 503 may further include memories that are remotely located relative to the processor 502, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Moreover, when one or more programs included in the above-mentioned computer device are executed by the one or more processors 502, one of the programs may perform the following operations:

Input the currently captured real-scene image frame into the preset target network model to obtain the actual image features of the real-scene image frame, and the target network model is determined based on the network model construction method provided in Embodiment 1 of the present invention; according to the set The image frame selection rule is to determine at least one image frame to be matched and the corresponding historical image features; based on the similarity value between the actual image features and each historical image feature, determine the closed-loop detection result of the real-scene image frame.

In addition, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the closed-loop detection method provided in the third embodiment of the present invention is implemented, wherein the method provided in the third embodiment above The method includes: inputting the currently captured real-scene image frame into a preset target network model to obtain the actual image features of the real-scene image frame, and the target network model is determined based on the network model construction method provided in Embodiment 1 of the present invention; according to The set image frame selection rule determines at least one image frame to be matched and the corresponding historical image features; based on the similarity value between the actual image features and each historical image feature, the closed-loop detection result of the real-scene image frame is determined.

Through the above description about the implementation mode, those skilled in the art can clearly understand that the present invention can be realized by means of software and necessary general-purpose hardware, and of course it can also be realized by hardware, but in many cases the former is a better implementation mode . Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a floppy disk of a computer , read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), flash memory (FLASH), hard disk or optical disc, etc., including several instructions to make a computer device (which can be a personal computer, A server, or a network device, etc.) executes the methods described in various embodiments of the present invention.

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention, and the present invention The scope is determined by the scope of the appended claims.

Claims (14)

1. A kind of 1. construction method of network model, it is characterised in that including：

   Topology information and configuration parameter information based on acquisition, structure form initial network model, wherein, the topology knot Structure information includes at least one of：Between the number of plies of convolutional layer, the number of plies of pond layer, the number of plies of full articulamentum and each layer Topology connection order；The configuration parameter information includes at least one of：The convolution step-length and convolution kernel of each convolutional layer are big The neuronal quantity of small and quantity, the pond step-length of each pond layer and pond window size and each full articulamentum；

   According to the training learning information of acquisition, initial network model described in repetitive exercise, obtain with standard weight data set Objective network model.
2. 2. according to the method for claim 1, it is characterised in that the convolutional layer includes 5 layers of convolutional layer, respectively volume 1 Lamination, the 2nd convolutional layer, the 3rd convolutional layer, the 4th convolutional layer and the 5th convolutional layer；

   The pond layer includes 2 layers, respectively the 1st pond layer and the 2nd pond layer；

   The full articulamentum includes 2 layers, respectively the 1st full articulamentum and the 2nd full articulamentum；

   The Topology connection order is expressed as：Input layer -- the 1st convolutional layer -- the 1st pond layer -- the 2nd convolutional layer -- the 2nd pond Layer -- the 3rd convolutional layer -- the 4th convolutional layer -- the 5th convolutional layer -- the 1st full articulamentum -- the 2nd full articulamentum -- output layer.
3. 3. according to the method for claim 2, it is characterised in that i-th convolutional layer includes the i-th _ 1 convolutional layer and the i-th _ 2 Convolutional layer, wherein, i value is 3,4 and 5, and the convolutional calculation of the i-th _ 1 convolutional layer and the i-th _ 2 convolutional layer is using victory Footpath connects；

   I-th _ j convolutional layers also include：I-th _ j_1 convolutional layers, i-th _ j_2 convolutional layers and i-th _ j_3 convolutional layers, Wherein, j value is 1 and 2.
4. 4. according to the method described in claim any one of 1-3, it is characterised in that it is described training learning information include it is following at least One of：The initial weight data and convolution letter of convolution kernel in input picture sample set, activation primitive, biased data, each convolutional layer The initial weight number of neuron in the initial weight data and pond function of several, each Chi Huacengzhongchiization window, each full articulamentum According to this and output category function；

   The standard weight data set includes at least one of：After repetitive exercise in each convolutional layer in convolution kernel, each pond layer Standard weight data corresponding to neuron in pond window and each full articulamentum.
5. A kind of 5. closed loop detection method, it is characterised in that including：

   The real scene image frame currently captured is inputted into default objective network model, obtains the real image of the real scene image frame Feature, the objective network model are determined based on the construction method described in claim any one of 1-4；

   According to the picture frame selection rule of setting, at least one picture frame to be matched of the real scene image frame is determined, and is obtained The historical image characteristic of each picture frame to be matched；

   Similarity value based on the real image feature Yu each historical image characteristic, determine the closed loop inspection of the real scene image frame Survey result.
6. 6. according to the method for claim 5, it is characterised in that the picture frame selection rule according to setting, determine institute At least one picture frame to be matched of real scene image frame is stated, and obtains the historical image characteristic of each characteristics of image to be matched, including：

   Obtain setting interval frame number and real scene image frame frame number, and by the frame number with it is described be spaced frame number difference It is defined as target frame number；

   Frame number is less than or equal to the history image frame of the target frame number as to be matched in the history information library built Picture frame；

   Obtain the historical image characteristic that each picture frame to be matched is determined based on the objective network model.
7. 7. according to the method for claim 5, it is characterised in that described to be based on the real image feature and each history image The Similarity value of feature, the closed loop testing result of the real scene image frame is determined, including：

   Calculate the Similarity value of the real image feature and each historical image characteristic；

   Picture frame to be matched corresponding to will be greater than setting the Similarity value of similar threshold value is defined as candidate's closed image frame, and by institute Candidate's closed image frame is stated added to candidate's closed loop collection of setting；

   If candidate's closed loop, which is concentrated, only has candidate's closed image frame, candidate's closed image frame is defined as institute State the closed loop region of real scene image frame；

   If candidate's closed loop, which is concentrated, has at least two candidate's closed image frames, the closed loop based on setting determines that strategy obtains Obtain the closed loop region of the real scene image frame.
8. 8. according to the method for claim 7, it is characterised in that the closed loop based on setting determines that strategy obtains the reality The closed loop region of scape picture frame, including：

   Under conditions of candidate's closed loop concentrates the frame number of candidate's closed image frame to be discrete, the real scene image frame is determined In the absence of closed loop region；

   Under conditions of candidate's closed loop is concentrated and the continuous candidate's closed image frame of frame number be present, determine that frame number continuously originates Frame number and end frame number, and based on the initial frame number to corresponding candidate's closed image frame history of forming image between the frame number of end Region, the history image region is defined as to the closed loop region of the real scene image frame.
9. A kind of 9. construction device of network model, it is characterised in that including：

   Initial construction module, for the topology information based on acquisition and configuration parameter information, structure forms initial network mould Type, wherein, the topology information includes at least one of：The number of plies of convolutional layer, the number of plies of pond layer, full articulamentum Topology connection order between the number of plies and each layer；The configuration parameter information includes at least one of：The volume of each convolutional layer The god of product step-length and convolution kernel size and number, the pond step-length of each pond layer and pond window size and each full articulamentum Through first quantity；

   Target determination module, for the training learning information according to acquisition, initial network model, is had described in repetitive exercise The objective network model of standard weight data set.
10. A kind of 10. loop detector, it is characterised in that including：

    Characteristic extracting module, the real scene image frame for will currently capture input default objective network model, obtain the reality The real image feature of scape picture frame, construction device of the objective network model described in based on claim any one of 13-14 It is determined that；

    Image chooses module, for the picture frame selection rule according to setting, determines that at least one of real scene image frame treats Picture frame is matched, and obtains the historical image characteristic of each picture frame to be matched；

    Determining module is detected, for the Similarity value based on the real image feature Yu each historical image characteristic, it is determined that described The closed loop testing result of real scene image frame.
11. A kind of 11. computer equipment, it is characterised in that including：

    One or more processors；

    Storage device, for storing one or more programs；

    One or more of programs are by one or more of computing devices so that one or more of processors are realized The construction method of network model as any one of claim 1-4.
12. 12. a kind of computer equipment, including：Camera, for capture images frame, it is characterised in that also include：

    One or more processors；

    Storage device, for storing one or more programs；

    One or more of programs are by one or more of computing devices so that one or more of processors are realized Closed loop detection method as any one of claim 5-8.
13. 13. a kind of computer-readable recording medium, is stored thereon with computer program, it is characterised in that the program is by processor The construction method of the network model as any one of claim 1-4 is realized during execution.
14. 14. a kind of computer-readable recording medium, is stored thereon with computer program, it is characterised in that the program is by processor The closed loop detection method as any one of claim 5-8 is realized during execution.