CN111680758A - Image training sample generation method and device - Google Patents

Abstract

The application discloses an image training sample generation method and device, and belongs to the field of computer vision. According to the method and the device, a plurality of three-dimensional transformation models can be obtained by performing model transformation on the obtained three-dimensional texture model of the target object, and one or more two-dimensional images corresponding to each three-dimensional transformation model are generated, so that a large number of image training samples similar to the target object can be obtained. That is, the scheme can quickly and automatically generate a large number of image training samples meeting the requirements, and saves labor and time cost. And because the image training samples in the scheme are obtained according to the three-dimensional texture model of the target object, the image training samples and the target object belong to the same type and have higher fidelity. Therefore, compared with an image training sample captured from the Internet, the method can simplify the sample processing process, reduce the problem of quality reduction of the sample introduced by manual operation, and is beneficial to subsequent training of a deep learning network.

Description

Image training sample generation method and device

technical field

The present application relates to the field of Computer Vision (CV), and in particular, to a method and apparatus for generating image training samples.

Background technique

At present, deep learning technology is widely used in the field of CV. In order to enhance the generalization ability of deep learning networks and reduce the overfitting of deep learning networks, it is usually necessary to use massive image training samples to train deep learning networks. Image training samples have become an urgent problem to be solved in the application of deep learning technology in the field of CV.

In the related art, image training samples can be obtained by a method of collecting images with a camera or grabbing images from the Internet. However, since the method of camera acquisition is limited by hardware and labor costs, it takes a long time to acquire images of sufficient magnitude. However, most of the images captured from the Internet cannot be used directly, and need to rely on manual sample enhancement work such as cleaning and labeling of the captured images. The time cost and labor cost are also high, and manual operation may lead to a decrease in sample quality. , which affects the training of deep learning networks.

SUMMARY OF THE INVENTION

The present application provides a method and device for generating image training samples, which can save the time cost and labor cost of acquiring image training samples, and can improve the quality of the samples. The technical solution is as follows:

In one aspect, a method for generating image training samples is provided, the method comprising:

Obtain the 3D texture model of the target object;

Perform model transformation on the three-dimensional texture model to obtain a plurality of three-dimensional transformation models, where the model transformation includes at least one of model deformation and texture transformation;

One or more 2D images corresponding to each 3D transformation model in the plurality of 3D transformation models are generated, and the generated 2D images are used as image training samples.

Optionally, the acquisition of the three-dimensional texture model of the target object includes:

creating a three-dimensional model of the target object;

According to the texture of the target object, texture mapping is performed on the three-dimensional model to obtain the three-dimensional texture model.

Optionally, when the model transformation includes the texture transformation, performing model transformation on the three-dimensional texture model to obtain a plurality of three-dimensional transformation models, including:

Semantic segmentation is performed on the three-dimensional texture model to obtain a plurality of local feature regions, and each local feature region in the plurality of local feature regions corresponds to a semantic;

According to the semantics corresponding to the multiple local feature regions, from the stored multiple texture materials, obtain multiple target texture materials corresponding to each local feature region;

According to a plurality of target texture materials corresponding to the plurality of local feature regions, texture transformation is performed on the three-dimensional texture model to obtain the plurality of three-dimensional transformation models.

Optionally, when the model transformation includes the model deformation, the model transformation is performed on the three-dimensional texture model to obtain a plurality of three-dimensional transformation models, including:

Get multiple groups of structure editing parameters;

According to the multiple sets of structure editing parameters, the geometric structures of the three-dimensional texture models are edited respectively to obtain the multiple three-dimensional transformation models.

Optionally, generating one or more two-dimensional images corresponding to each of the three-dimensional transformation models in the plurality of three-dimensional transformation models includes:

Get reference camera parameters;

According to the reference camera parameters, each 3D transformation model is projected from one or more different viewing directions to obtain one or more 2D images corresponding to the corresponding 3D transformation model.

In another aspect, an image training sample generating apparatus is provided, the apparatus comprising:

an acquisition module for acquiring the 3D texture model of the target object;

a transformation module, configured to perform model transformation on the three-dimensional texture model to obtain a plurality of three-dimensional transformation models, where the model transformation includes at least one of model deformation and texture transformation;

A generating module is configured to generate one or more two-dimensional images corresponding to each three-dimensional transform model in the plurality of three-dimensional transform models, and use the generated two-dimensional images as image training samples.

Optionally, the obtaining module includes:

Create a submodule for creating a three-dimensional model of the target object;

The mapping sub-module is configured to perform texture mapping on the three-dimensional model according to the texture of the target object to obtain the three-dimensional texture model.

Optionally, when the model transformation includes the texture transformation, the transformation module includes:

a semantic segmentation sub-module, configured to perform semantic segmentation on the three-dimensional texture model to obtain a plurality of local feature regions, and each local feature region in the plurality of local feature regions corresponds to a semantic;

a first acquisition sub-module, configured to acquire a plurality of target texture materials corresponding to each local feature region from a plurality of stored texture materials according to the semantics corresponding to the plurality of local feature regions;

The transformation sub-module is configured to perform texture transformation on the three-dimensional texture model according to multiple target texture materials corresponding to the multiple local feature regions respectively, to obtain the multiple three-dimensional transformation models.

Optionally, when the model transformation includes the model deformation, the transformation module includes:

The second acquisition sub-module is used to acquire multiple groups of structure editing parameters;

The editing sub-module is configured to edit the geometric structure of the three-dimensional texture model according to the plurality of sets of structural editing parameters to obtain the plurality of three-dimensional transformation models.

Optionally, the generation module includes:

The third acquisition sub-module is used to acquire the reference camera parameters;

The projection sub-module is used for projecting each 3D transformation model from one or more different viewing angles according to the reference camera parameters to obtain one or more 2D images corresponding to the corresponding 3D transformation model.

In another aspect, a computer device is provided, the computer device includes a processor, a communication interface, a memory and a communication bus, and the processor, the communication interface and the memory communicate with each other through the communication bus , the memory is used for storing a computer program, and the processor is used for executing the program stored in the memory, so as to realize the steps of the above-mentioned image training sample generation method.

In another aspect, a computer-readable storage medium is provided, and a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned image training sample generation method are implemented.

In another aspect, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the steps of the above-described image training sample generation method.

The technical solution provided by this application can at least bring the following beneficial effects:

By performing model transformation on the acquired 3D texture model of the target object, multiple 3D transformation models can be obtained, and then one or more 2D images corresponding to each 3D transformation model can be generated. Image training samples. That is, the solution provided in this application can quickly and automatically generate a large number of image training samples that meet the needs. Compared with the actual camera collection samples, it saves equipment investment, labor costs and time costs, and can generate tens of thousands of times in the same time. even more samples. And since the image training samples in this solution are obtained according to the three-dimensional texture model of the target object, these image training samples belong to the same type as the target object and have high fidelity. In this way, compared with the image training samples captured from the Internet, the sample processing process can be simplified, thereby reducing the problem of sample quality degradation caused by manual operations during the sample processing process, which is beneficial to the subsequent training of the deep learning network.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a flowchart of a method for generating an image training sample provided by an embodiment of the present application;

2 is a schematic diagram of a three-dimensional model of a target object created in an embodiment of the present application;

3 is a schematic diagram of a texture of a target object collected by an embodiment of the present application;

4 is a schematic diagram of a three-dimensional texture model of a target object obtained in an embodiment of the present application;

5 is a schematic diagram of a three-dimensional transformation model obtained by performing model deformation on the three-dimensional texture model shown in FIG. 4 in an embodiment of the present application;

6 is a schematic diagram of obtaining a two-dimensional image by projecting a three-dimensional transformation model according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an apparatus for generating an image training sample provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a computer device provided by an embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

At present, deep learning technology is widely used in the field of CV. In order to enhance the generalization ability of deep learning networks and reduce the overfitting of deep learning networks, it is usually necessary to use massive image training samples to train deep learning networks. Image training samples have become an urgent problem to be solved in the application of deep learning technology in the field of CV. With the technical solutions provided by the embodiments of the present application, a large number of image training samples that meet the requirements can be quickly generated. For example, quickly generating a large number of human image training samples can be used to train a deep learning network to obtain a deep learning model that can be used for character recognition, etc., for example, a large number of car image training samples can be generated for The deep learning network can be trained to obtain a deep learning model that can be used for vehicle detection. For another example, a large number of image training samples of interior decoration can be generated, which can be used to train the deep learning network to obtain deep learning that can be used for environmental structure detection, etc. Model.

Next, the method for generating image training samples provided by the embodiments of the present application will be explained in detail.

FIG. 1 is a flowchart of a method for generating an image training sample provided by an embodiment of the present application. Please refer to FIG. 1 , the method includes the following steps.

Step 101: Acquire a three-dimensional texture model of the target object.

In the embodiment of the present application, in order to generate a large number of image training samples of specified types, one or more two-dimensional images of the target object can be collected first, and the three-dimensional texture model of the target object can be obtained by a three-dimensional reconstruction method. The specified type can be specified according to actual needs, and the target object is an actual object of the specified type. For example, if a deep learning network needs to be trained for vehicle recognition, the specified type can be a vehicle, and the target object can be an actual car. .

It should be noted that, in this embodiment of the present application, the specified type may be any object type, such as a human body, a vehicle, a table and chair, a landscape, an interior decoration, and the like.

In some embodiments, after collecting one or more two-dimensional images of the target object, a three-dimensional model of the target object may be created first, and then texture mapping is performed on the three-dimensional model according to the texture of the target object to obtain a three-dimensional texture model .

In this embodiment of the present application, a three-dimensional reconstruction algorithm can be used to reconstruct the three-dimensional model of the target object, that is, to reconstruct the three-dimensional geometric structure of the target object. For example, a three-dimensional model can be reconstructed from the contours of the target object. The 3D reconstruction algorithm may be multi-view stereo geometry (Multi-View Stereo, MVS) reconstruction, contour-based shape recovery (Shape from Silhouette, SfS), etc., which is not limited in this embodiment of the present application.

After the three-dimensional model of the target object is created, texture mapping can be performed on the corresponding regions of the three-dimensional model according to the acquired textures of various regions of the target object to obtain a three-dimensional texture model of the target object.

For example, assuming that the target object is a car, after the three-dimensional model of the car is created, texture mapping can be performed on the three-dimensional model of the car according to the collected texture of the car to obtain the three-dimensional texture model of the car.

For another example, assuming that the target object is a person, FIG. 2 is a three-dimensional model created according to a plurality of two-dimensional images of the target object collected, FIG. 3 is a plurality of textures of the target object collected, and FIG. The texture shown in Figure 2 is mapped to the corresponding area of the 3D model shown in Figure 2, and the 3D texture model of the target object is obtained.

In other embodiments, the three-dimensional texture model of the target object can be obtained directly according to one or more two-dimensional images of the target object. For example, one or more 2D images of the target object can be input into an end-to-end deep learning model, which can directly output a 3D texture model of the target object.

In some other embodiments, a specified type of 3D texture model can be obtained from a 3D texture model library. For example, a 3D texture model of a car can be obtained from a 3D texture model library.

In the embodiment of the present application, in addition to collecting one or more two-dimensional images of the target object to reconstruct the three-dimensional texture model of the target object, one or more depth images of the target object and the texture of the target object may also be collected. , performing three-dimensional reconstruction on the acquired depth image to obtain a three-dimensional model of the target object, and then performing texture mapping on the three-dimensional model according to the texture of the target object to obtain a three-dimensional texture model of the target object. The method for performing 3D reconstruction on the depth image may be a 3D reconstruction method based on deep learning, or may be other methods, which are not limited in this application.

Alternatively, the 3D point cloud of the target object can also be collected by a laser device, the texture of the target object can be collected by a camera, and then 3D reconstruction of the collected 3D point cloud can be performed to obtain a 3D model of the target object, and then the 3D model of the target object can be obtained according to the texture of the target object. 3D model for texture mapping. The method for performing 3D reconstruction on a 3D point cloud may be a 3D reconstruction method based on deep learning, or may be other methods, which are not limited in this application.

Step 102: Perform model transformation on the three-dimensional texture model to obtain a plurality of three-dimensional transformation models. The model transformation includes at least one of model deformation and texture transformation.

In this embodiment of the present application, after acquiring the three-dimensional texture model of the target object, model transformation may be performed on the three-dimensional texture model, wherein the model transformation includes at least one of model deformation and texture transformation. Next, four implementation manners for performing model transformation on a three-dimensional texture model provided by the embodiments of the present application will be introduced.

In the first implementation manner, when the model transformation includes texture transformation, the three-dimensional texture model can be semantically segmented to obtain multiple local feature regions, and each local feature region in the multiple local feature regions corresponds to a semantic. Then, according to the semantics corresponding to the multiple local feature regions, from the stored multiple texture materials, multiple target texture materials corresponding to each local feature region are obtained. Then, according to multiple target texture materials corresponding to multiple local feature regions, texture transformation is performed on the 3D texture model to obtain multiple 3D transformation models.

In this embodiment of the present application, only the three-dimensional texture model of the target object may be subjected to texture transformation to obtain multiple three-dimensional transformation models. Exemplarily, first, semantic segmentation may be performed on the 3D texture model to obtain a plurality of corresponding local feature regions with semantics, that is, the 3D texture model is divided into a plurality of local feature regions with different semantics. Among them, the method of semantic feature extraction may be deep learning-based semantic feature extraction, semantic segmentation, model parameterization, etc., or may be other semantic segmentation methods.

It should be noted that, in this embodiment of the present application, the 3D texture model of the target object can be directly semantically segmented. For example, the 3D texture model of the human body can be directly semantically segmented to obtain local feature regions such as hair, face, and limbs. These regions on the 3D texture model are semantically labeled, so as to obtain a plurality of corresponding semantic local feature regions. In addition, in the embodiment of the present application, the three-dimensional texture model of the target object can also be projected to multiple viewing directions to obtain multiple two-dimensional images, and then these two-dimensional images are semantically segmented to obtain multiple corresponding semantic local features The corresponding regions of the three-dimensional texture model are semantically marked according to the multiple local feature regions. For example, you can first project the 3D texture model of a car to 6 different viewing directions to obtain 6 different 2D images, and then perform semantic segmentation on the 6 2D images respectively to obtain multiple corresponding semantic images. Then, according to the multiple local feature regions, the corresponding regions of the three-dimensional texture model are semantically marked, so that each local feature region will have a corresponding semantic.

After semantic segmentation is performed to obtain multiple local feature regions corresponding to semantics, multiple target textures corresponding to each local feature region can be obtained from the stored multiple texture materials according to the semantics corresponding to the multiple local feature regions. material.

In this embodiment of the present application, a plurality of texture materials may be stored on a computer device, and the stored texture materials may be stored according to semantics, and one or more texture materials are stored corresponding to each semantic. Eyebrows, desktop, leather and other semantics are stored. When the semantics is desktop, there can be correspondingly stored texture materials of multiple different desktops. The color, size, highlight, texture pattern, brightness, etc. of each desktop can be different.

In this embodiment of the present application, multiple target texture materials corresponding to each local feature region may be acquired from the stored multiple texture materials according to the semantics corresponding to the plurality of local feature regions determined above. For any local feature region In other words, the semantics of the acquired multiple target texture materials may be the same as the semantics of the local feature area, or may be related to the semantics of the local feature area.

It should be noted that the quantity of target texture materials corresponding to any two acquired local feature regions may be the same or different. In addition, for any local feature area, all target texture materials with the same semantics or relatedness may be obtained from the stored multiple texture materials, or a target texture material not exceeding a preset number may be obtained.

After acquiring multiple target texture materials corresponding to each local feature area, you can use the multiple target texture materials corresponding to the multiple local feature areas respectively to perform texture transformation on the corresponding area of the 3D texture model to obtain multiple 3D transformations Model.

Among them, for each local feature region, a texture transformation can be performed on the local feature region by using each target texture material in the obtained corresponding multiple target texture materials to obtain a texture transformation of the local feature region, so that , a variety of texture transformations of the local feature area can be obtained. In this embodiment of the present application, each texture transformation of any one of the multiple local feature regions can be arbitrarily combined with multiple texture transformations of the remaining local feature regions except the local feature region to obtain Multiple 3D transformation models.

Exemplarily, it is assumed that the 3D texture model has 5 local feature regions corresponding to semantics, and each local feature region can correspond to 3 target texture materials, so that the 5th power of 3 can be obtained by combination, that is, 243 A three-dimensional transformation model. Suppose it is determined that the 3D texture model has 4 local feature regions corresponding to semantics, and these 4 local feature regions correspond to 2, 3, 3, and 5 target texture materials respectively, so that 2 × 3 × 3 × 5 can be combined to obtain = 90 3D transformation models.

In the second implementation manner, when the model transformation includes model deformation, multiple sets of structural editing parameters can be obtained first, and then the geometric structure of the 3D texture model can be edited according to the multiple sets of structural editing parameters to obtain multiple 3D transformation models. .

In the embodiment of the present application, only the three-dimensional texture model of the target object may be subjected to various model deformations to obtain a plurality of three-dimensional transformation models.

In the embodiment of the present application, multiple sets of structural editing parameters can be preset, and each set of structural editing parameters can be used to edit the geometric structure of the three-dimensional texture model once to obtain a three-dimensional transformation model. For example, according to any set of structural editing parameters, the vertex coordinates, normal direction, patch topology, etc. of the 3D texture model can be changed to obtain a 3D transformation model.

The method for model deformation may be an optical flow method, an iterative closest point, parameterized template driving, or other methods, and may also be other methods, which are not limited in this embodiment of the present application.

Exemplarily, after performing model deformation on the three-dimensional texture model shown in FIG. 4 according to a set of structural editing parameters, a three-dimensional transformation model as shown in FIG. 5 can be obtained.

In the third implementation manner, when the model transformation includes model deformation and texture transformation, the three-dimensional texture model can be texture transformed to obtain multiple texture transformation models, and then each texture transformation model in the multiple texture transformation models can be obtained. Perform model deformation to obtain a three-dimensional transformation model corresponding to each texture transformation model.

In this embodiment of the present application, the three-dimensional texture model may be subjected to texture transformation and then model deformation to obtain more three-dimensional transformed models. The methods of texture transformation and model deformation may refer to the above-mentioned related introductions, which will not be repeated here.

In the fourth implementation manner, when the model transformation includes model deformation and texture transformation, the 3D texture model can be first subjected to model deformation to obtain multiple 3D deformation models, and then each 3D deformation model in the multiple 3D deformation models can be obtained. Perform texture transformation to obtain a 3D transformation model corresponding to each 3D deformation model.

In this embodiment of the present application, the three-dimensional texture model can be deformed and then texture transformed to obtain more three-dimensional transformed models. The methods of texture transformation and model deformation may refer to the above-mentioned related introductions, which will not be repeated here.

Step 103: Generate one or more two-dimensional images corresponding to each of the three-dimensional transformation models in the plurality of three-dimensional transformation models, and use the generated two-dimensional images as image training samples.

In this embodiment of the present application, after obtaining multiple 3D transformation models, each 3D transformation model can be projected to one or more different viewing directions to generate corresponding one or more 2D images, and the generated 2D transformation models can be Images are used as image training samples.

In this embodiment of the present application, reference camera parameters may be obtained, and each 3D transformation model may be projected from one or more different viewing angles according to the reference camera parameters, to obtain one or more 2D transformation models corresponding to the corresponding 3D transformation model. image. The reference camera parameters may refer to the camera parameters used when creating the 3D texture model, or may be other camera parameters specified by the user.

It should be noted that, in order to ensure the fidelity of the 2D image obtained by projection, the projection of the 3D transformation model is the projection performed by simulating the camera shooting, that is, the 3D transformation model is projected according to the reference camera parameters, and the obtained The fidelity of 2D images is high. The reference camera parameters may include one or more groups of camera parameters, and each group of camera parameters may include camera intrinsic parameters, distortion coefficients, image resolution, viewing angle range, and the like. In addition, the projected viewing angle direction may be one or more, and each viewing angle direction may be the direction of a ray passing through a camera viewpoint. When there are multiple viewing angle directions for projection, each viewing angle direction can correspond to a set of camera parameters. Since a set of camera parameters can determine a viewing angle range, each viewing angle direction also corresponds to a viewing angle range, and each viewing angle direction corresponds to The viewing angle ranges may overlap or not overlap, and the sizes of the viewing angle ranges corresponding to each viewing angle direction may be the same or different.

It should be noted that the viewing angle range corresponding to the corresponding viewing angle direction can be determined according to the camera internal parameters and image resolution in the reference camera parameters. Based on this, in this embodiment of the present application, each viewing angle range may or may not be specified.

Exemplarily, as shown in FIG. 6 , it is assumed that there are six perspective directions from the six camera viewpoints shown in the figure to the center point of the 3D transformation model, the six camera viewpoints shown in the figure are located on the same horizontal plane, and the six viewpoints are The viewing angle range corresponding to the direction is divided into 360 degrees of the horizontal plane, that is, the viewing angle range corresponding to each viewing angle direction is 60 degrees, and the six viewing angle ranges do not overlap. Projection, you can get 6 different two-dimensional images. In this embodiment of the present application, the number of viewing angle directions, the position of the camera viewpoint, the line of sight direction, the camera internal parameters, the distortion coefficient, and the image resolution can be specified according to requirements. For example, a viewing angle direction may also be a direction from top to bottom, and the corresponding viewing angle range may be specified as 30 degrees or 120 degrees.

In the embodiment of the present application, the deformation of the model can generate various models of different three-dimensional geometric structures, which greatly expands the number of image training samples from the dimensions of the three-dimensional geometric structures. Texture transformation can perform semantic structure analysis on 3D texture models, replace corresponding areas with texture materials with the same semantics, and perform combined transformations of various texture replacements, which greatly expands the number of image training samples from the dimension of texture content. Finally, a two-dimensional image is generated according to the camera projection method, and the two-dimensional image is used as an image training sample. In this way, the generated image training sample has high fidelity, is more cost-effective than actual camera samples, and can flexibly respond to user needs. .

To sum up, in the embodiment of the present application, by performing model transformation on the acquired 3D texture model of the target object, multiple 3D transformation models can be obtained, and then one or more 2D images corresponding to each 3D transformation model are generated. , in this way, a large number of training samples of images similar to the target object can be obtained. That is, the solution provided in this application can quickly and automatically generate a large number of image training samples that meet the needs. Compared with the actual camera collection samples, it saves equipment investment, labor costs and time costs, and can generate tens of thousands of times in the same time. even more samples. And since the image training samples in this solution are obtained according to the three-dimensional texture model of the target object, these image training samples belong to the same type as the target object and have high fidelity. In this way, compared with the image training samples captured from the Internet, the sample processing process can be simplified, thereby reducing the problem of sample quality degradation caused by manual operations during the sample processing process, which is beneficial to the subsequent training of the deep learning network.

After the sample library is constructed based on the 3D texture model of the target object in the above manner, these samples can be used to train the deep learning network, and the trained deep learning network can be used to recognize faces, vehicles, and actions. Since the two-dimensional image samples generated by the three-dimensional texture model are closer to the real object, and the capture process is saved, the training cost is lower, and the trained deep learning network has a higher recognition accuracy.

7 is a schematic structural diagram of an image training sample generating apparatus provided by an embodiment of the present application. The image training sample generating apparatus may be implemented as part or all of computer equipment by software, hardware, or a combination of the two. Referring to FIG. 7 , the apparatus includes: an acquisition module 701 , a transformation module 702 and a generation module 703 .

an acquisition module 701, configured to acquire a three-dimensional texture model of a target object;

A transformation module 702, configured to perform model transformation on the three-dimensional texture model to obtain multiple three-dimensional transformation models, where the model transformation includes at least one of model deformation and texture transformation;

The generating module 703 is configured to generate one or more 2D images corresponding to each 3D transformation model in the multiple 3D transformation models, and use the generated 2D images as image training samples.

Optionally, the obtaining module 701 includes:

Create a submodule for creating a 3D model of the target object;

The mapping sub-module is used to perform texture mapping on the 3D model according to the texture of the target object to obtain a 3D texture model.

Optionally, when the model transformation includes texture transformation, the transformation module 702 includes:

The semantic segmentation sub-module is used to perform semantic segmentation on the three-dimensional texture model to obtain multiple local feature regions, and each local feature region in the multiple local feature regions corresponds to a semantic;

The first acquisition sub-module is configured to acquire multiple target texture materials corresponding to each local feature region from the stored multiple texture materials according to the semantics corresponding to the multiple local feature regions;

The transformation sub-module is used for performing texture transformation on the 3D texture model according to the multiple target texture materials corresponding to the multiple local feature regions respectively to obtain multiple 3D transformation models.

Optionally, when the model transformation includes model deformation, the transformation module 702 includes:

The second acquisition sub-module is used to acquire multiple groups of structure editing parameters;

The editing sub-module is used to edit the geometric structure of the 3D texture model according to multiple sets of structural editing parameters to obtain multiple 3D transformation models.

Optionally, the generating module 703 includes:

The third acquisition sub-module is used to acquire reference camera parameters, and the reference camera parameters refer to the camera parameters used when creating the 3D texture model;

The projection sub-module is used to project each 3D transformation model from one or more different viewing angles according to the reference camera parameters to obtain one or more 2D images corresponding to the corresponding 3D transformation model.

In the embodiment of the present application, by performing model transformation on the acquired 3D texture model of the target object, multiple 3D transformation models can be obtained, and then one or more 2D images corresponding to each 3D transformation model can be generated. In this way, one can obtain A large number of training samples of images similar to the target object. That is, the solution provided in this application can quickly and automatically generate a large number of image training samples that meet the needs. Compared with the actual camera collection samples, it saves equipment investment, labor costs and time costs, and can generate tens of thousands of times in the same time. even more samples. And since the image training samples in this solution are obtained according to the three-dimensional texture model of the target object, these image training samples belong to the same type as the target object and have high fidelity. In this way, compared with the image training samples captured from the Internet, the sample processing process can be simplified, thereby reducing the problem of sample quality degradation caused by manual operations during the sample processing process, which is beneficial to the subsequent training of the deep learning network.

It should be noted that when the image training sample generating apparatus provided in the above embodiment generates image training samples, only the division of the above functional modules is used as an example. Module completion means dividing the internal structure of the device into different functional modules to complete all or part of the functions described above. In addition, the image training sample generating apparatus provided in the above embodiments and the image training sample generating method embodiments belong to the same concept, and the specific implementation process thereof is detailed in the method embodiments, which will not be repeated here.

FIG. 8 is a structural block diagram of a computer device 800 provided by an embodiment of the present application. The computer device 800 may be a smart phone, a tablet computer, a notebook computer or a desktop computer, and the like.

Generally, computer device 800 includes: processor 801 and memory 802 .

The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 801 may adopt at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, programmable logic array) accomplish. The processor 801 may also include a main processor and a coprocessor. The main processor is a processor used to process data in a wake-up state, also called a CPU (Central Processing Unit, central processing unit); A low-power processor for processing data in a standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor 801 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is used to process computing operations related to machine learning.

Memory 802 may include one or more computer-readable storage media, which may be non-transitory. Memory 802 may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, a non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 801 to implement the image training provided by the method embodiments in this application. Sample generation method.

In some embodiments, the computer device 800 may also optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected by a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 803 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804 , a touch display screen 805 , a camera 806 , an audio circuit 807 , a positioning component 808 and a power supply 809 .

The peripheral device interface 803 may be used to connect at least one peripheral device related to I/O (Input/Output) to the processor 801 and the memory 802 . In some embodiments, processor 801, memory 802, and peripherals interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one of processor 801, memory 802, and peripherals interface 803 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The radio frequency circuit 804 is used for receiving and transmitting RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals. The radio frequency circuit 804 communicates with the communication network and other communication devices via electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. Radio frequency circuitry 804 may communicate with other computer devices through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: World Wide Web, Metropolitan Area Network, Intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), wireless local area network and/or WiFi (Wireless Fidelity, Wireless Fidelity) network. In some embodiments, the radio frequency circuit 804 may further include a circuit related to NFC (Near Field Communication, short-range wireless communication), which is not limited in this application.

The display screen 805 is used to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 also has the ability to acquire touch signals on or above the surface of the display screen 805 . The touch signal can be input to the processor 801 as a control signal for processing. At this time, the display screen 805 may also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards. In some embodiments, the display screen 805 may be one disposed on the front panel of the computer device 800; in other embodiments, the display screen 805 may be at least two disposed on different surfaces of the computer device 800 or in a folded design; In other embodiments, display screen 805 may be a flexible display screen disposed on a curved or folded surface of computer device 800 . Even, the display screen 805 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 805 can be made of materials such as LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light emitting diode).

The camera assembly 806 is used to capture images or video. Optionally, the camera assembly 806 includes a front camera and a rear camera. Usually, the front camera is arranged on the front panel of the computer device, and the rear camera is arranged on the back of the computer device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth-of-field camera to realize the background blur function, the main camera It is integrated with the wide-angle camera to achieve panoramic shooting and VR (Virtual Reality, virtual reality) shooting functions or other integrated shooting functions. In some embodiments, camera assembly 806 may also include a flash. The flash can be a single color temperature flash or a dual color temperature flash. Dual color temperature flash refers to the combination of warm light flash and cold light flash, which can be used for light compensation under different color temperatures.

Audio circuitry 807 may include a microphone and speakers. The microphone is used to collect the sound waves of the user and the environment, convert the sound waves into electrical signals, and input them to the processor 801 for processing, or to the radio frequency circuit 804 to realize voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively disposed in different parts of the computer device 800 . The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 801 or the radio frequency circuit 804 into sound waves. The loudspeaker can be a traditional thin-film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for distance measurement and other purposes. In some embodiments, audio circuitry 807 may also include a headphone jack.

The positioning component 808 is used to locate the current geographic location of the computer device 800 to implement navigation or LBS (Location Based Service). The positioning component 808 may be a positioning component based on the GPS (Global Positioning System, global positioning system) of the United States, the Beidou system of China, or the Galileo system of Russia.

Power supply 809 is used to power various components in computer device 800 . The power source 809 may be alternating current, direct current, disposable batteries or rechargeable batteries. When the power source 809 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. Wired rechargeable batteries are batteries that are charged through wired lines, and wireless rechargeable batteries are batteries that are charged through wireless coils. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, computer device 800 also includes one or more sensors 810 . The one or more sensors 810 include, but are not limited to, an acceleration sensor 811 , a gyro sensor 812 , a pressure sensor 813 , a fingerprint sensor 814 , an optical sensor 815 , and a proximity sensor 816 .

The acceleration sensor 811 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the computer device 800 . For example, the acceleration sensor 811 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 801 can control the touch display screen 805 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 811 . The acceleration sensor 811 can also be used for game or user movement data collection.

The gyroscope sensor 812 can detect the body direction and rotation angle of the computer device 800 , and the gyroscope sensor 812 can cooperate with the acceleration sensor 811 to collect the 3D actions of the user on the computer device 800 . The processor 801 can implement the following functions according to the data collected by the gyroscope sensor 812 : motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

The pressure sensor 813 may be disposed on the side frame of the computer device 800 and/or on the lower layer of the touch display screen 805 . When the pressure sensor 813 is disposed on the side frame of the computer device 800 , the user's holding signal of the computer device 800 can be detected, and the processor 801 can perform left and right hand identification or shortcut operations according to the holding signal collected by the pressure sensor 813 . When the pressure sensor 813 is disposed on the lower layer of the touch display screen 805 , the processor 801 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 805 . The operability controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

The fingerprint sensor 814 is used to collect the user's fingerprint, and the processor 801 identifies the user's identity according to the fingerprint collected by the fingerprint sensor 814 , or the fingerprint sensor 814 identifies the user's identity according to the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 801 authorizes the user to perform related sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. Fingerprint sensor 814 may be provided on the front, back, or side of computer device 800 . When the computer device 800 is provided with physical buttons or a manufacturer's logo, the fingerprint sensor 814 can be integrated with the physical buttons or the manufacturer's logo.

Optical sensor 815 is used to collect ambient light intensity. In one embodiment, the processor 801 may control the display brightness of the touch display screen 805 according to the ambient light intensity collected by the optical sensor 815 . Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 805 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 805 is decreased. In another embodiment, the processor 801 may also dynamically adjust the shooting parameters of the camera assembly 806 according to the ambient light intensity collected by the optical sensor 815 .

Proximity sensor 816 , also referred to as a distance sensor, is typically provided on the front panel of computer device 800 . Proximity sensor 816 is used to collect the distance between the user and the front of computer device 800 . In one embodiment, when the proximity sensor 816 detects that the distance between the user and the front of the computer device 800 gradually decreases, the processor 801 controls the touch display screen 805 to switch from the bright screen state to the off screen state; when the proximity sensor 816 When it is detected that the distance between the user and the front of the computer device 800 gradually increases, the processor 801 controls the touch display screen 805 to switch from the off-screen state to the bright-screen state.

Those skilled in the art can understand that the structure shown in FIG. 8 does not constitute a limitation to the computer device 800, and may include more or less components than the one shown, or combine some components, or adopt different component arrangements.

In some embodiments, a computer-readable storage medium is also provided, and a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the image training sample generation method in the above-mentioned embodiments are implemented. For example, the computer-readable storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

It should be noted that the computer-readable storage medium mentioned in this application may be a non-volatile storage medium, in other words, may be a non-transitory storage medium.

It should be understood that, all or part of the steps of implementing the above embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions may be stored in the computer-readable storage medium described above.

That is, in some embodiments, there is also provided a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the steps of the above-described image training sample generation method.

The above-mentioned examples provided for this application are not intended to limit this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection scope of this application. Inside.

Claims (10)

1. An image training sample generation method, characterized in that the method comprises:

acquiring a three-dimensional texture model of a target object;

performing model transformation on the three-dimensional texture model to obtain a plurality of three-dimensional transformation models, wherein the model transformation comprises at least one of model deformation and texture transformation;

and generating one or more two-dimensional images corresponding to each three-dimensional transformation model in the plurality of three-dimensional transformation models, and taking the generated two-dimensional images as image training samples.

2. The method of claim 1, wherein obtaining the three-dimensional texture model of the target object comprises:

creating a three-dimensional model of the target object;

and performing texture mapping on the three-dimensional model according to the texture of the target object to obtain the three-dimensional texture model.

3. The method of claim 1, wherein when the model transformation comprises the texture transformation, the model transforming the three-dimensional texture model to obtain a plurality of three-dimensional transformation models comprises:

performing semantic segmentation on the three-dimensional texture model to obtain a plurality of local feature regions, wherein each local feature region in the plurality of local feature regions corresponds to a semantic;

according to the semantics corresponding to the local feature regions, acquiring a plurality of target texture materials corresponding to each local feature region from a plurality of stored texture materials;

and performing texture transformation on the three-dimensional texture model according to a plurality of target texture materials corresponding to the plurality of local characteristic regions respectively to obtain a plurality of three-dimensional transformation models.

4. The method of claim 1, wherein when the model transformation comprises the model deformation, the model transforming the three-dimensional texture model to obtain a plurality of three-dimensional transformation models comprises:

acquiring a plurality of groups of structure editing parameters;

and editing the geometric structures of the three-dimensional texture models respectively according to the multiple groups of structure editing parameters to obtain the multiple three-dimensional transformation models.

5. The method of any of claims 1-4, wherein generating one or more two-dimensional images corresponding to each of the plurality of three-dimensional transformation models comprises:

acquiring reference camera parameters;

and projecting each three-dimensional transformation model from one or more different view angle directions according to the reference camera parameters to obtain one or more two-dimensional images corresponding to the corresponding three-dimensional transformation models.

6. An image training sample generation apparatus, characterized in that the apparatus comprises:

the acquisition module is used for acquiring a three-dimensional texture model of the target object;

the transformation module is used for carrying out model transformation on the three-dimensional texture model to obtain a plurality of three-dimensional transformation models, and the model transformation comprises at least one of model deformation and texture transformation;

and the generating module is used for generating one or more two-dimensional images corresponding to each three-dimensional transformation model in the plurality of three-dimensional transformation models and taking the generated two-dimensional images as image training samples.

7. The apparatus of claim 6, wherein the obtaining module comprises:

a creating submodule for creating a three-dimensional model of the target object;

and the mapping submodule is used for performing texture mapping on the three-dimensional model according to the texture of the target object to obtain the three-dimensional texture model.

8. The apparatus of claim 6, wherein when the model transform comprises the texture transform, the transform module comprises:

the semantic segmentation submodule is used for performing semantic segmentation on the three-dimensional texture model to obtain a plurality of local feature areas, and each local feature area in the plurality of local feature areas corresponds to a semantic;

the first obtaining submodule is used for obtaining a plurality of target texture materials corresponding to each local characteristic region from a plurality of stored texture materials according to the semantics corresponding to the local characteristic regions;

and the transformation submodule is used for carrying out texture transformation on the three-dimensional texture model according to a plurality of target texture materials respectively corresponding to the local feature areas to obtain a plurality of three-dimensional transformation models.

9. The apparatus of claim 6, wherein when the model transformation comprises the model deformation, the transformation module comprises:

the second obtaining submodule is used for obtaining a plurality of groups of structure editing parameters;

and the editing submodule is used for respectively editing the geometric structures of the three-dimensional texture models according to the multiple groups of structure editing parameters to obtain the multiple three-dimensional transformation models.

10. The apparatus according to any one of claims 6-9, wherein the generating means comprises:

the third acquisition sub-module is used for acquiring reference camera parameters;

and the projection submodule is used for projecting each three-dimensional transformation model from one or more different view angle directions according to the reference camera parameters to obtain one or more two-dimensional images corresponding to the corresponding three-dimensional transformation models.

Images