CN109035380B - Face modification method, device and equipment based on three-dimensional reconstruction and storage medium - Google Patents

Abstract

The invention provides a face modification method, a face modification device, face modification equipment and a storage medium based on three-dimensional reconstruction, and belongs to the technical field of image processing. The method comprises the following steps: determining a target modification area from an original face image; carrying out three-dimensional reconstruction on the target modification area to obtain three-dimensional space information corresponding to the target modification area; and coloring the target modification area according to the three-dimensional space information to obtain a modified face image. According to the invention, the target modification area is colored according to the three-dimensional spatial information, so that the texture sense is stronger, the details of the target modification area can be effectively restored, the texture is finer, the stereoscopic impression can be effectively enhanced by polishing the target modification area in the normal direction, and the user experience is further remarkably improved.

Description

Face modification method, device, equipment and storage medium based on 3D reconstruction

technical field

The present invention relates to the field of image processing, in particular, to a face modification method, device, equipment and storage medium based on three-dimensional reconstruction.

Background technique

With the development of science and technology and the improvement of the application level of technology industrialization, the performance of mobile phones is getting better and better, and the hardware configuration has become more and more complete. But at the same time, with the increasingly fierce competition in the mobile phone market, competing hardware configurations can no longer attract more electronic consumers. Therefore, most mobile phone manufacturers are pursuing differentiated functional planning, design, and marketing of mobile phone products. For example, for the application scenario of lipstick modification in face modification, the current existing lipstick modification is through face recognition on the original image; when the original image contains a face area, locate the lip area from the face area, And according to the lip area, the average lip color data is obtained; according to the average lip color data and the color temperature data of the original image, the pixel points of the lip area are processed. Because the processing is based on the two-dimensional lip area, there are technical problems in the prior art that the processing results depend on the average lip color data, the processing results are not delicate, the texture is not strong, there is no real three-dimensional effect, and the user experience is poor.

Contents of the invention

The face modification method, device, equipment and storage medium based on three-dimensional reconstruction provided by the embodiments of the present invention can solve the problems in the prior art that the processing result does not have three-dimensional gloss, relies on the average lip color data, the processing result is not delicate, and the texture feels Not strong, no real three-dimensional sense, poor user experience technical problems.

In order to achieve the above object, the technical solution adopted in the embodiment of the present invention is as follows:

In the first aspect, an embodiment of the present invention provides a face modification method based on three-dimensional reconstruction, including: determining a target modification area from an original face image; performing three-dimensional reconstruction on the target modification area to obtain the target modification area Corresponding three-dimensional space information; performing coloring processing on the target modification area according to the three-dimensional space information to obtain a modified human face image.

In combination with the first aspect, the embodiment of the present invention provides a first possible implementation of the first aspect, the determining the target modification area from the original face image includes: according to the preset face key point detection model from the original Determining face key point information in the face image; separating a region to be modified from the original face image according to the face key point information, and the region to be modified is the target modification region.

In combination with the first possible implementation manner of the first aspect, the embodiment of the present invention provides a second possible implementation manner of the first aspect, performing three-dimensional reconstruction on the target modification area to obtain the target modification area The corresponding three-dimensional space information includes: inputting the target modification area and the face key point information into a preset three-dimensional basic model, and outputting the three-dimensional space information corresponding to the target modification area.

With reference to the second possible implementation manner of the first aspect, the embodiment of the present invention provides a third possible implementation manner of the first aspect, the inputting the target modification area and the facial key point information into the preset A three-dimensional basic model is set, and the three-dimensional space information corresponding to the target modification area is output, including: obtaining the basic parameters corresponding to the face key point information and the target modification area, and the weight coefficients corresponding to the basic parameters; The basic parameters determine a preset three-dimensional basic model matching the basic parameters; the preset three-dimensional basic model is weighted by the weight coefficient to obtain three-dimensional space information corresponding to the target modification area.

With reference to the first possible implementation of the first aspect, the embodiment of the present invention provides a fourth possible implementation of the first aspect, which further includes: acquiring image sample data marked with key points of human faces; The data is used to perform neural network training on the initial face key point detection model, and a preset face key point detection model is obtained.

In combination with the first aspect, the embodiment of the present invention provides a fifth possible implementation manner of the first aspect, wherein the target modification area is colored according to the three-dimensional space information to obtain a modified face image, It includes: performing coloring processing on the target modification area according to the three-dimensional space information to obtain a three-dimensional texture coloring result; replacing the three-dimensional texture coloring result on the target modification area of the original face image to obtain a modified face images.

With reference to the fifth possible implementation manner of the first aspect, the embodiment of the present invention provides a sixth possible implementation manner of the first aspect, performing coloring processing on the target modification area according to the three-dimensional space information, Obtaining the 3D texture coloring result includes: determining the 3D subdivision grid corresponding to the target modification area according to the 3D space information; migrating the color corresponding to the coordinates of the texture to the coordinate position corresponding to the 3D subdivision grid to obtain 3D texture shader result.

With reference to the sixth possible implementation manner of the first aspect, the embodiment of the present invention provides a seventh possible implementation manner of the first aspect, wherein the determination of the 3D profile corresponding to the target modification area according to the 3D space information Subdividing the grid includes: performing triangulation on the three-dimensional space information to obtain a three-dimensional subdivision grid corresponding to the target modification area.

With reference to the seventh possible implementation manner of the first aspect, the embodiment of the present invention provides an eighth possible implementation manner of the first aspect, wherein the triangulation of the three-dimensional space information is performed to obtain the target modified The 3D subdivision grid corresponding to the region includes: performing the nearest neighbor non-intersection triangulation on the 3D spatial information to obtain the 3D subdivision grid corresponding to the target modification region.

With reference to the fifth possible implementation of the first aspect, this embodiment of the present invention provides a ninth possible implementation of the first aspect, the replacement of the 3D texture coloring result with the original face image Obtaining a modified human face image on the target modification area includes: performing feathering processing on an edge area of the target modification area to obtain a modified human face image.

In the second aspect, the embodiment of the present invention provides a face modification device based on three-dimensional reconstruction, including: a modification target determination unit, configured to determine the target modification area from the original face image; a first processing unit, configured to performing three-dimensional reconstruction on the target modification area to obtain three-dimensional space information corresponding to the target modification area; the second processing unit is used to perform coloring processing on the target modification area according to the three-dimensional space information to obtain a modified human face image.

In a third aspect, a terminal device provided by an embodiment of the present invention includes: a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor executes the computer program When realizing the steps of the face modification method based on three-dimensional reconstruction as described in any one of the first aspect.

In a fourth aspect, an embodiment of the present invention provides a storage medium, where instructions are stored on the storage medium, and when the instructions are run on a computer, the computer is made to execute the method based on any one of the first aspect. Face Retouching Method for 3D Reconstruction.

Compared with the prior art, the embodiments of the present invention bring the following beneficial effects:

In the face modification method, device, device and storage medium based on three-dimensional reconstruction provided by the embodiments of the present invention, the target modification area is determined from the original face image, and the target modification area is three-dimensionally reconstructed to obtain the target modification area The corresponding three-dimensional space information can effectively restore the details of the target modification area and enhance the three-dimensional effect of the target modification area, and color the target modification area according to the three-dimensional space information, which can make the texture of the target modification area Stronger and finer textures further enhance the three-dimensionality of the target retouching area, thereby significantly improving the user experience.

Other features and advantages of the present disclosure will be set forth in the following description, or some of the features and advantages can be inferred or unambiguously determined from the description, or can be known by implementing the above-mentioned techniques of the present disclosure.

In order to make the above-mentioned objects, features and advantages of the present invention more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and thus It should be regarded as a limitation on the scope, and those skilled in the art can also obtain other related drawings based on these drawings without creative work.

Fig. 1 is the flow chart of the face modification method based on 3D reconstruction provided by the first embodiment of the present invention;

Fig. 2 is the original face image in the face modification method based on three-dimensional reconstruction shown in Fig. 1;

Fig. 3 is the face detection result schematic diagram in the face modification method based on three-dimensional reconstruction shown in Fig. 1;

FIG. 4 is a schematic diagram of the principle of triangulation in the face modification method based on three-dimensional reconstruction shown in FIG. 1;

Fig. 5 is the lip region image in the face modification method based on three-dimensional reconstruction shown in Fig. 1;

Fig. 6 is a schematic diagram of the lip area triangulation results in the face modification method based on three-dimensional reconstruction shown in Fig. 1;

Fig. 7 is a schematic diagram of functional modules of a face modification device based on 3D reconstruction provided by the second embodiment of the present invention;

Fig. 8 is a schematic diagram of a terminal device provided by a third embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention. Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Some embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

first embodiment

Due to the existing technical problems of relying on the average lip color data, the processing results are not delicate, the texture is not strong, there is no real three-dimensional effect, and the user experience is poor, in order to solve the above technical problems, this embodiment firstly provides a A face modification method based on three-dimensional reconstruction, it should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and, although shown in the flow chart logical order, but in some cases the steps shown or described may be performed in an order different from that presented here. This embodiment will be described in detail below.

Please refer to FIG. 1 , which is a flowchart of a face modification method based on 3D reconstruction provided by an embodiment of the present invention. The specific process shown in FIG. 1 will be described in detail below.

Step S101, determining the target retouching area from the original face image.

Optionally, the original face image may be a face image stored by the user in a terminal device (such as a mobile phone or a tablet). It may also be the preview data frame obtained by starting the preview video stream through an image acquisition device (such as a camera). Or it is a face image downloaded by the user through the network.

Wherein, the target modification area is one or more areas in the original face image, for example, the target modification area may be but not limited to lip area, cheekbone area, eyebrow area or eyelid area and so on.

As an implementation, step S101 includes: performing face detection on the original face image to obtain face key point information; collecting user's request instruction and extracting the target modification area from the face key point information according to the request instruction.

Wherein, the request instruction may be a request for a lip area, or an eye area or an eyebrow area, and the like.

Optionally, the user's request instruction can be monitored through a click event of the terminal device or the request instruction input by the user through the terminal device can be obtained.

For example, face recognition is performed on the original face image as shown in Figure 2 by a face recognition method (for example, a recognition algorithm based on facial feature points or an algorithm for identifying using a neural network), and the result shown in Figure 3 is obtained. The point set in the displayed face image (that is, the key point information of the face). Determine the target modification area, such as the lip area, from the point set according to the user's request instruction.

As another implementation, step S101 includes: determining face key point information from the original face image according to a preset face key point detection model; separating face key point information from the original face image according to the face key point information The area to be modified is selected, and the area to be modified is the target modification area.

Optionally, by separating the target modification area from the original face image (such as copying the target modification area), image processing is performed on the target modification area separately, so that the processed image data is smaller, and then can be Effectively reduce data processing pressure and improve data processing efficiency.

Optionally, the target modification area can be processed directly, that is, the modification of the target modification area is directly completed on the original face image.

Optionally, it also includes: acquiring image sample data marked with key points of human face; performing neural network training on the initial key point detection model of human face through the sample image data to obtain a preset key point detection model of human face.

Among them, the face key point information may be but not limited to face contour points, eye contour points, nose contour points, eyebrow contour points, forehead contour points, upper lip contour points, lower lip contour points, etc.

Optionally, it is possible to obtain images that have been marked with key points of human faces in advance as image sample data, or by collecting a preset number of human face images (for example, the preset number can be 100,000), and then for each Annotate key points of the face on the face image to obtain annotation data, and an image containing the annotation data is used as image sample data.

Optionally, performing neural network training on the initial facial key point detection model through the image sample data to obtain a preset human face key point detection model, including: dividing the image sample data into a training set according to a preset ratio, Verification set and test set; Carry out neural network training to initial face key point detection model by described training set, and verify (real-time adjustment training parameter) to intermediate result in training process with described verification set; When training accuracy and verification When the accuracy reaches the preset threshold, the training process is stopped to obtain the trained facial key point detection model; the trained human face key point detection model is tested according to the test set to obtain the test result; if the The test result satisfies the preset rule (for example, performance or ability), and the trained facial key point detection model is used as the preset human face key point detection model.

Wherein, the preset ratio can be set according to actual needs. Generally, the proportion of the training set is larger than that of the verification set and the test set. For example, the ratio of training set, validation set and test set can be 8:1:1.

Among them, the training accuracy refers to the error range obtained by training the facial key point detection model through the training set. Generally, a preset threshold is set for the training accuracy in advance, and the size of the preset threshold can be set according to actual needs. Here, no specific limitation is made. In the same way, the verification accuracy refers to a value obtained when verifying the trained face key point detection model, and a preset threshold is also set for the verification accuracy in advance, and the size of the preset threshold can be set according to actual needs , and is not specifically limited here.

As an application scenario, it is assumed that the target modification area is lips, and lipstick needs to be applied to the lips. Then the user first runs the program of the face modification method based on three-dimensional reconstruction provided by the embodiment of the present invention, and the program loads the default parameter mapping table of lipstick (different lip shapes have different three-dimensional reconstruction parameters, three-dimensional meshing parameters, etc., such as: petal lip, small round lip, cherry lip, etc.), among which, the user can also adjust the corresponding parameter size by himself. Then execute the face modification method based on 3D reconstruction provided by the embodiment of the present invention to modify the target modification area (that is, to achieve the effect of applying lipstick to the lips).

Step S102, performing three-dimensional reconstruction on the target modification area to obtain three-dimensional space information corresponding to the target modification area.

The three-dimensional space information includes the target point set corresponding to the target modification area, and the three-dimensional coordinates and Euler angles of the target point set in the three-dimensional space system.

As an implementation manner, step S102 includes: inputting the target modification area and the face key point information into a preset three-dimensional basic model, and outputting three-dimensional space information corresponding to the target modification area.

Optionally, inputting the target modification area and the face key point information into a preset three-dimensional basic model, and outputting the three-dimensional space information corresponding to the target modification area includes: obtaining the face key points Information and basic parameters corresponding to the target modification area and weight coefficients corresponding to the basic parameters; determining a preset three-dimensional basic model matching the basic parameters according to the basic parameters; The 3D basic model is set to carry out weighting processing to obtain the 3D spatial information corresponding to the target modification area.

Wherein, the basic parameter refers to the coordinates of all points in the target point set, that is, the three-dimensional space coordinate point set, which includes the three-dimensional coordinates of each point and its Euler angle.

The weight coefficient is the weight of each basic parameter.

Optionally, the weight coefficients may be generated in real time.

Optionally, performing weighting processing on the preset three-dimensional basic models according to the weight coefficients includes: multiplying each preset three-dimensional basic model with its corresponding weight coefficients to obtain a product, and adding all the products together.

For example, assuming that there are 3 preset 3D basic models, namely G1, G2, and G3, and their corresponding weight coefficients are Q1, Q2, and Q3, the result of the weighting process is G1*Q1+G2*Q2+ G3*Q3.

In this embodiment, a preset number of preset three-dimensional basic models (such as 100) are established in advance, such as preset three-dimensional basic models such as mouth opening, mouth closing, left corner of the mouth rising, and right corner of the mouth rising.

Optionally, acquiring the basic parameters corresponding to the face key point information and the target modification area includes: detecting the target modification area according to the human face key point information, and obtaining shape parameters corresponding to the target modification area (such as , lip shape when opening the mouth) and expression parameters (for example, smile), the shape parameters and expression parameters are decomposed to obtain the basic parameters.

Among them, the shape parameters and expression parameters are the results calculated through the coordinates of a point in a specific area (such as lips or face area), such as the coordinates of the corners of the mouth relative to the preset 3D model (for example, the expressionless basic model ) of the mouth, if its longitudinal direction is reduced, then it can be abstracted as a smile. For another example, if the distance between the upper and lower lips is larger than that of the expressionless basic model, then it can be abstracted as an open mouth.

Optionally, decomposing the shape parameters and the expression parameters to obtain the basic parameters includes: decomposing according to the key points corresponding to the shape parameters and the expression parameters to obtain the basic parameters.

Of course, in actual use, other methods can also be used for decomposing, for example, decomposing shape parameters and expression parameters into multiple basic parameters according to coordinates. Or decompose the shape parameters and expression parameters into multiple basic parameters through the preset area.

Step S103, performing coloring processing on the target modification area according to the three-dimensional space information to obtain a modified human face image.

As an implementation, step S103 includes: performing coloring processing on the target modification area according to the three-dimensional space information to obtain a three-dimensional texture coloring result; replacing the three-dimensional texture coloring result with the original face image On the target modified area, the modified face image is obtained.

Optionally, performing coloring processing on the target modification area according to the three-dimensional space information to obtain a three-dimensional texture coloring result, including: determining a three-dimensional subdivision grid corresponding to the target modification area according to the three-dimensional space information; The above three-dimensional subdivision mesh is used for texture coloring processing, and the three-dimensional texture coloring result is obtained.

Optionally, performing texture coloring processing on the three-dimensional subdivision grid includes: migrating the color corresponding to the coordinates of the texture to the coordinate position corresponding to the three-dimensional subdivision grid, so as to realize the coloring process.

Optionally, determining the 3D subdivision grid corresponding to the target modification area according to the 3D space information includes: performing triangulation on the 3D space information to obtain the 3D subdivision grid corresponding to the target modification area . Specifically, acquire the target point set in the three-dimensional space information and the three-dimensional information corresponding to the target point set, perform triangulation on the target point set according to the three-dimensional information and a triangulation algorithm (delaunay), and obtain the target The 3D subdivision mesh corresponding to the grooming area.

Optionally, performing triangulation on the three-dimensional space information (that is, performing triangulation on the target point set according to the three-dimensional information of the target point set through a triangulation algorithm (delaunay)), to obtain the corresponding The three-dimensional subdivision grid includes: performing the nearest-neighbor non-intersection triangulation on the three-dimensional space information through a triangulation algorithm to obtain the three-dimensional subdivision grid corresponding to the target modification area.

In this embodiment, through the non-intersection triangulation, each triangle in the obtained non-intersection triangulation can have no intersection, so that when coloring, the coordinates corresponding to a certain triangle will not be caused It is repeatedly colored to make the coloring more uniform, which in turn makes the coloring effect better.

Of course, in actual use, the nearest neighbor non-intersection triangulation may also be performed on the three-dimensional spatial information in other ways. For example, the nearest-neighbor intersection-free triangulation of the three-dimensional space information can be realized by algorithms such as divide-and-conquer method, point-by-point insertion method (Lawson algorithm), or triangular network generation method (BowyerWatson algorithm).

For example, as shown in FIG. 4 , the target point set is triangulated according to the triangulation principle to obtain a grid composed of multiple triangles, that is, a three-dimensional subdivision grid.

For example, taking the lip area as the target modification area as an example, the point set in the lip shown in Figure 5 is triangulated, so as to obtain the three-dimensional subdivision mesh corresponding to the target modification area (ie, the lip area), that is, The schematic diagram shown in Figure 6.

Optionally, the three-dimensional subdivision grid corresponding to the target modification area may also be obtained by performing rectangular subdivision (for example, quadrilateral or polygon) on the three-dimensional spatial information.

In this embodiment, by performing texture coloring on the 3D subdivided mesh, the sense of texture is stronger, and the details of the target modification area can be effectively restored, and the texture is finer. By performing normal lighting on the target modification area, it can Effectively enhance the three-dimensional sense.

Optionally, replacing the three-dimensional texture coloring result on the target modification area of the original face image includes: performing feathering processing on an edge area of the target modification area to obtain a modified face image.

In this embodiment, after independently modifying the target modification area separated from the original human face image, the modified three-dimensional texture coloring result is mapped (or replaced) on the target modification area in the original human face image .

In a possible embodiment, after performing feathering processing on the edge area of the target modification area to obtain the modified face image, the method further includes: displaying the face image on a terminal device.

For example, assuming that the target modification area is lips, the lip area is first separated from the original face image, and then the lip area is colored to obtain a three-dimensional texture coloring result, and the three-dimensional texture coloring result is mapped on the original face image on the lip area to realize the modification of the face image. Therefore, on the one hand, by separating out the target modification area and modifying the target modification area, the pressure of data processing can be effectively reduced and the efficiency of data processing can be improved; The area corresponding to the face image, and then complete the modification of the face, making the modification effect better.

The face modification method based on 3D reconstruction provided by the embodiment of the present invention determines the target modification area from the original face image, performs 3D reconstruction on the target modification area, and obtains the 3D spatial information corresponding to the target modification area , so that the details of the target modification area can be effectively restored and the three-dimensional effect of the target modification area can be enhanced, and the target modification area can be colored according to the three-dimensional space information, which can make the texture of the target modification area stronger and more delicate. The three-dimensional effect of the target modification area is further enhanced, thereby significantly improving user experience.

second embodiment

Corresponding to the face modification method based on 3D reconstruction in the first embodiment, Fig. 7 shows the face modification device based on 3D reconstruction using the face modification method based on 3D reconstruction shown in the first embodiment in one-to-one correspondence . As shown in FIG. 7 , the three-dimensional reconstruction-based face modification device 400 includes a modification target determination unit 410 , a first processing unit 420 , a second processing unit 430 and a third processing unit 440 . Wherein, the realization functions of the modification target determination unit 410, the first processing unit 420, the second processing unit 430, and the third processing unit 440 are in one-to-one correspondence with the corresponding steps in the first embodiment. To avoid redundant description, this embodiment does not Describe in detail.

A modification target determining unit 410, configured to determine a target modification area from the original face image.

Optionally, the original face image is a preview image.

Optionally, the modified target determination unit 410 is configured to determine face key point information from the original face image according to a preset face key point detection model; The region to be modified is isolated, and the region to be modified is the target modification region.

Optionally, it also includes acquiring image sample data labeled with key points of human face; and performing neural network training on the initial key point detection model of human face through the sample image data to obtain a preset key point detection model of human face.

The first processing unit 420 is configured to perform three-dimensional reconstruction on the target modification area to obtain three-dimensional space information corresponding to the target modification area.

Optionally, the first processing unit 420 is further configured to input the target modification area and the face key point information into a preset 3D basic model, and output the 3D space information corresponding to the target modification area.

Optionally, inputting the target modification area and the face key point information into a preset three-dimensional basic model, and outputting the three-dimensional space information corresponding to the target modification area includes: obtaining the face key points Information and basic parameters corresponding to the target modification area and weight coefficients corresponding to the basic parameters; determine a preset three-dimensional basic model matching the basic parameters according to the basic parameters; The 3D basic model is set to carry out weighting processing to obtain the 3D spatial information corresponding to the target modification area.

The second processing unit 430 is configured to perform coloring processing on the target modification area according to the three-dimensional space information to obtain a modified human face image.

Optionally, the second processing unit 430 is further configured to perform coloring processing on the target modification area according to the three-dimensional space information to obtain a three-dimensional texture coloring result; replace the three-dimensional texture coloring result into the original face image On the target modification area, the modified face image is obtained.

Optionally, performing coloring processing on the target modification area according to the three-dimensional space information to obtain a three-dimensional texture coloring result, including: determining a three-dimensional subdivision grid corresponding to the target modification area according to the three-dimensional space information; The above three-dimensional subdivision mesh is used for texture coloring processing, and the three-dimensional texture coloring result is obtained.

Optionally, the determining the 3D subsection grid corresponding to the target modification area according to the 3D space information includes: performing triangulation on the 3D space information to obtain the 3D subsection grid corresponding to the target modification area. sub-grid.

Optionally, performing triangulation on the three-dimensional space information to obtain the three-dimensional subdivision grid corresponding to the target modification area includes: performing nearest-neighbor intersection-free triangulation on the three-dimensional space information, A three-dimensional subdivision grid corresponding to the target modification area is obtained.

Optionally, replacing the three-dimensional texture coloring result on the target modification area of the original face image includes: performing feathering processing on an edge area of the target modification area to obtain a modified face image.

Optionally, after the second processing unit 430, the three-dimensional reconstruction-based face modification apparatus 400 further includes a fourth processing unit, configured to display the modified face image on a display terminal.

third embodiment

As shown in FIG. 8 , it is a schematic diagram of a terminal device 300 . The terminal device 300 includes a memory 302, a processor 304, a computer program 303 stored in the memory 302 and operable on the processor 304, and a display 305 for displaying a modified face image, the When the computer program 303 is executed by the processor 304, the method for modifying a face based on 3D reconstruction in the first embodiment is realized, and to avoid repetition, details are not repeated here. Alternatively, when the computer program 303 is executed by the processor 304, the functions of each model/unit in the three-dimensional reconstruction-based face modification device described in the second embodiment are realized, and details are not repeated here to avoid repetition.

Exemplarily, the computer program 303 can be divided into one or more modules/units, and one or more modules/units are stored in the memory 302 and executed by the processor 304 to implement the present invention. One or more modules/units may be a series of computer program instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the execution process of the computer program 303 in the terminal device 300 . For example, the computer program 303 can be divided into the modification target determining unit 410, the first processing unit 420 and the second processing unit 430 in the second embodiment, and the specific functions of each unit are as described in the first embodiment or the second embodiment , which will not be repeated here.

The terminal device 300 may be computing devices such as desktop computers, notebooks, palmtop computers, and cloud servers.

Wherein, the memory 302 can be, but not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read only memory (Programmable Read-Only Memory, PROM), can Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc. Wherein, the memory 302 is used to store a program, and the processor 304 executes the program after receiving an execution instruction, and the process definition method disclosed in any embodiment of the foregoing embodiments of the present invention can be applied to the processor 304, or implemented by processor 304 .

The processor 304 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 304 can be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (NetworkProcessor, NP), etc.; it can also be a digital signal processor (Digital Signal Processor, DSP), a dedicated integrated Circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The display 305 can be an LCD display or an LED display. For example, the display screen on a mobile phone.

It can be understood that the structure shown in FIG. 8 is only a schematic structural diagram of the terminal device 300, and the terminal device 300 may also include more or fewer components than those shown in FIG. 8 . Each component shown in FIG. 8 may be implemented by hardware, software or a combination thereof.

Fourth embodiment

An embodiment of the present invention also provides a storage medium, on which instructions are stored. When the instructions are run on a computer, the computer program is executed by a processor to implement the three-dimensional based on the first embodiment. In order to avoid repetition, the reconstructed face modification method will not be repeated here. Alternatively, when the computer program is executed by the processor, the functions of each model/unit in the 3D reconstruction-based face modification device of the second embodiment are implemented, and to avoid repetition, details are not repeated here.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by hardware, or by means of software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention It can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.), and include several instructions to make a computer device (which can be It is a personal computer, a server, or a network device, etc.) to execute the method of each implementation scenario of the present invention.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention. It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

Claims (10)

1. A face modification method based on three-dimensional reconstruction is characterized by comprising the following steps:

determining a target modification area from an original face image;

performing three-dimensional reconstruction on the target modification region to obtain three-dimensional space information corresponding to the target modification region;

coloring the target modification area according to the three-dimensional space information to obtain a modified human face image;

wherein, the determining the target modification region from the original face image comprises: determining face key point information from an original face image according to a preset face key point detection model; separating a region to be modified from the original face image according to the face key point information, wherein the region to be modified is the target modification region;

the three-dimensional reconstruction of the target modification region to obtain three-dimensional space information corresponding to the target modification region includes: acquiring the human face key point information, basic parameters corresponding to the target modification area and used for representing a three-dimensional space coordinate point set of the target modification area, and weight coefficients corresponding to the basic parameters; determining a preset three-dimensional basic model matched with the basic parameters according to the basic parameters; and carrying out weighting processing on the preset three-dimensional basic model through the weighting coefficient to obtain three-dimensional space information corresponding to the target modification area.

2. The method of claim 1, further comprising:

acquiring image sample data marked with key points of the face;

and carrying out neural network training on the initial face key point detection model through the image sample data to obtain a preset face key point detection model.

3. The method according to claim 1, wherein the coloring the target modified region according to the three-dimensional spatial information to obtain a modified face image comprises:

coloring the target modification area according to the three-dimensional space information to obtain a three-dimensional texture coloring result;

and replacing the three-dimensional texture coloring result on the target modification area of the original face image to obtain a modified face image.

4. The method according to claim 3, wherein the rendering the target modified region according to the three-dimensional spatial information to obtain a three-dimensional texture rendering result comprises:

determining a three-dimensional subdivision grid corresponding to the target modification area according to the three-dimensional space information;

and transferring the color of the coordinate corresponding to the texture to the coordinate position corresponding to the three-dimensional subdivision grid to obtain a three-dimensional texture coloring result.

5. The method according to claim 4, wherein the determining the three-dimensional split mesh corresponding to the target modification region according to the three-dimensional spatial information comprises:

and triangulating the three-dimensional space information to obtain a three-dimensional subdivision grid corresponding to the target modification area.

6. The method according to claim 5, wherein the triangulating the three-dimensional spatial information to obtain a three-dimensional mesh corresponding to the target modification region comprises:

and performing nearest non-cross triangulation on the three-dimensional space information to obtain a three-dimensional mesh corresponding to the target modification area.

7. The method according to claim 3, wherein said replacing the three-dimensional texture coloring result onto the target modified region of the original face image to obtain a modified face image, comprises:

and performing feathering processing on the edge area of the target modification area to obtain a modified face image.

8. A face modification device based on three-dimensional reconstruction is characterized by comprising:

a modified target determining unit for determining a target modified region from the original face image;

the first processing unit is used for carrying out three-dimensional reconstruction on the target modification area to obtain three-dimensional space information corresponding to the target modification area;

the second processing unit is used for carrying out coloring processing on the target modification area according to the three-dimensional space information to obtain a modified human face image;

the modification target determining unit is further specifically configured to determine face key point information from an original face image according to a preset face key point detection model; separating a region to be modified from the original face image according to the face key point information, wherein the region to be modified is the target modification region;

the first processing unit is further specifically configured to obtain the face key point information, a basic parameter corresponding to the target modification region and used for representing a three-dimensional space coordinate point set of the target modification region, and a weight coefficient corresponding to the basic parameter; determining a preset three-dimensional basic model matched with the basic parameters according to the basic parameters; and carrying out weighting processing on the preset three-dimensional basic model through the weighting coefficient to obtain three-dimensional space information corresponding to the target modification area.

9. A terminal device, comprising: a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the three-dimensional reconstruction based face embellishment method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium having stored thereon instructions which, when run on a computer, cause the computer to execute the method of face embellishment based on three-dimensional reconstruction of any of claims 1 to 7.

Images