CN116311279A - Generation of sample images, model training, character recognition method, equipment and medium - Google Patents

Abstract

The present invention relates to the technical field of image processing, in particular to the generation of sample images, model training, character recognition methods, devices and media. The generation method includes acquiring target text and style features in a target scene and first sample characters, target text and The style feature is obtained by encoding the text style of the real character image in the target scene; based on the target diffusion model, the first sample character, the target text and the style feature are processed to obtain the sample character image in the target scene, and the target diffusion model uses The font style of the first sample character is transferred to the real character image to generate the sample character image; the sample character image is spliced with the background image in the target scene to obtain the sample image. The font style of the first sample character is transferred to the real character image by using the target diffusion model, so that the style is unified and a realistic sample image is obtained.

Description

Generation of sample images, model training, character recognition method, equipment and medium

technical field

The present invention relates to the technical field of image processing, in particular to methods, equipment and media for generation of sample images, model training, and character recognition.

Background technique

Optical character recognition (OCR) has many applications in the field of deep learning applications that are rapidly developing at this stage. It has excellent performance in applications such as document and certificate recognition. In terms of machine vision, character recognition requirements in different scenarios have emerged accordingly. . In the case of sufficient sample data, according to the two-stage mode of the trained character recognition model, the text content on the picture can be well recognized. Based on this, the sample data affects the accuracy of the character recognition model.

However, in some application scenarios, the amount of sample data is small, and the fidelity of the sample data obtained through some data enhancement processing is low. For example, for character recognition on clothing, in order to increase the amount of sample data, the existing solution is to directly paste characters on the body. Even if some text and image amplification and transformation are added, it is difficult to make it look like the characters that come with the clothes. Instead of pasting it later, resulting in a lower fidelity of the sample image.

Contents of the invention

In view of this, the embodiments of the present invention provide a generation of sample images, model training, character recognition, equipment and media, so as to solve the problem of low fidelity of sample images.

According to the first aspect, an embodiment of the present invention provides a method for generating a sample image, including:

Acquiring target text and style features and first sample characters in the target scene, where the target text and style features are obtained by text style encoding of real character images in the target scene;

Process the first sample character, the target text and style features based on a target diffusion model to obtain a sample character image in the target scene, and the target diffusion model is used to convert the first sample character The font style is transferred to the real character image to generate the sample character image;

The sample character image is spliced with the background image in the target scene to obtain a sample image.

The sample image generation method provided by the embodiment of the present invention uses the target text and style features obtained from the real character image in the target scene as the conditions of the target diffusion model, and uses the target diffusion model to migrate the font style of the first sample character to the real character image. In the character image, so as to ensure that the text and image in the generated sample character image can achieve a unified style and be closer to the real text style of the target scene, and then splicing the sample character image with the background image in the target scene can obtain a realistic sample character image.

In some implementation manners, said acquiring the target text and style features in the target scene includes:

Acquiring the real character image;

Carry out image feature extraction to described real character image, obtain image style coding;

Carrying out text feature extraction to the text content in described real character image, obtains text coding;

The image style coding and the text coding are fused to obtain the target text and style features.

The sample image generation method provided by the embodiment of the present invention extracts image features and text features respectively for real character images, and then fuses the two, so that the obtained target text and style features include both image features and text features. There are textual features that improve the reliability of the resulting target text and style features.

In some embodiments, the fusion of the image style code and the text code to obtain the target text and style features includes:

performing attention processing on the image style code and the text code to obtain the attention between the image style code and the text code;

The attention is fused with the text code, and the fusion result is processed by a feed-forward network to obtain the target text and style features.

The sample image generation method provided by the embodiment of the present invention realizes the fusion of image style coding and text coding through attention, so that different text contents focus on different parts of a given image style coding, and improves the reliability of the obtained fusion results And the authenticity, on this basis, after the processing of the feedforward network, richer semantic features can be extracted, which further ensures the authenticity of the real character image text style encoding, so that the target text and style features can be used as the input of the target diffusion model When the conditions are met, the target diffusion model can obtain a more realistic sample image by diffusion.

In some embodiments, the splicing of the sample character image and the background image in the target scene to obtain a sample image includes:

Acquiring a background image in the target scene, and identifying a region of interest on the background image to obtain an image of a region of interest;

Rotating the sample character image at any angle to obtain a rotated sample character image;

The rotated sample character image is spliced with the ROI image to obtain the sample image.

The sample image generation method provided by the embodiment of the present invention, through the identification of the region of interest, facilitates the subsequent splicing of the sample character image to the region of interest, so as to be close to the use in the target scene; at the same time, the sample character image is rotated at any angle , so that a large number of sample images can be generated, thereby enriching the number of sample images.

In some embodiments, the training method of the target diffusion model includes:

Acquiring the text and style features of the second sample character and the sample image in the target scene;

During the forward diffusion process of the preset diffusion model, a noise image is generated based on the second sample character;

Based on the noise image and the text and style features of the sample image in the target scene, the reverse diffusion process of the preset diffusion model is trained to determine the target diffusion model.

The sample image generation method provided by the embodiment of the present invention uses the second sample character as supervision, and the text and style features of the sample image in the target scene as conditions to train the reverse diffusion process of the preset diffusion model, which can ensure that the obtained The target diffusion model is able to generate images that fit the target scene.

According to the second aspect, the embodiment of the present invention also provides a method for training a character detection model, including:

Acquiring a sample image, the sample image is obtained according to the method for generating a sample image described in the first aspect of the present invention or in any implementation manner of the first aspect;

Acquiring label data of the sample image, the label data including the position information of the text content in the sample image and the target rotation angle;

Inputting the sample image into the character detection model to obtain predicted position information and predicted rotation angle of the text content in the sample image;

Based on the predicted position information, the predicted rotation angle and the label data, the parameters of the character detection model are updated to obtain a target character detection model.

The character detection model training method provided by the embodiment of the present invention can improve the accuracy of the obtained target character detection model by performing character detection model training on the basis of a large number of realistic sample images obtained in the target scene. ; and the trained target character detection model also outputs the rotation angle of the text content to improve the detection accuracy of the target character detection model. In addition, in subsequent character recognition, the detected text content can be rotated by using the rotation angle, so as to further improve the accuracy of character recognition.

According to the third aspect, the embodiment of the present invention also provides a method for training a character recognition model, including:

Acquiring a sample image, the sample image is obtained according to the method for generating a sample image described in the first aspect of the present invention or any implementation manner of the first aspect;

Input the sample image into the target character detection model to obtain the position and rotation angle of the text content in the sample image, and the target character detection model is obtained by training according to the character detection model training method described in the second aspect of the present invention of;

using the position of the text content and the rotation angle to rotate the text content in the sample image to obtain the target text content;

Based on the target text content and the text label of the sample image, the parameters of the character recognition model are updated to obtain the target character recognition model.

In the character recognition model training method provided by the embodiment of the present invention, before inputting the detected text lines into the character recognition model, the text content is rotated and corrected using the predicted position and rotation angle to ensure the obtained target text content The angles are unified, and then the parameters of the character recognition model are updated by using the target text content with a unified angle, which further improves the accuracy of the trained target character recognition model.

According to the fourth aspect, the embodiment of the present invention also provides a character recognition method, including:

Obtain the image to be processed under the target scene;

Input the image to be processed into the target character detection model to obtain the position and rotation angle of the text content in the image to be processed, and the target character detection model is a training method for the character detection model according to the second aspect of the present invention obtained by training;

Using the position and rotation angle of the text content in the image to be processed to rotate the text content in the image to be processed to obtain the text content to be recognized;

Inputting the text content to be recognized into the target character recognition model to obtain the character recognition result of the image to be processed, the target character recognition model is obtained by training according to the character recognition model training method described in the third aspect of the present invention .

The character recognition method provided by the embodiment of the present invention, since the target character detection model and the target character recognition model are trained with a large number of realistic sample images, have high character detection and character recognition accuracy. The character recognition model performs character recognition on the image to be processed, and can obtain more accurate character recognition results.

According to the fifth aspect, the embodiment of the present invention also provides a sample image generation module, including:

The first acquisition module is used to acquire the target text and style feature in the target scene and the font style of the first sample character, and the target text and style feature are obtained by encoding the text style of the real character image in the target scene of;

A style processing module, configured to process the font style of the first sample character and the target text and style features based on a target diffusion model to obtain a sample character image in the target scene, and the target diffusion model is used for Migrating the font style of the first sample character to the real character image to generate the sample character image;

A splicing module, configured to splice the sample character image with the background image in the target scene to obtain a sample image.

According to the sixth aspect, the embodiment of the present invention also provides a character detection model training device, including:

The second acquisition module is used to acquire a sample image, the sample image is obtained according to the method for generating a sample image described in the first aspect of the present invention or any one of the implementations of the first aspect;

A third acquiring module, configured to acquire tag data of the sample image, where the tag data includes position information of text content in the sample image and a target rotation angle;

A first prediction module, configured to input the sample image into the character detection model to obtain predicted position information and predicted rotation angle of text content in the sample image;

A first updating module, configured to update parameters of the character detection model based on the predicted position information, the predicted rotation angle, and the label data, so as to obtain a target character detection model.

According to the seventh aspect, the embodiment of the present invention also provides a character recognition model training device, including:

A fourth acquisition module, configured to acquire a sample image, the sample image is obtained according to the method for generating a sample image described in the first aspect of the present invention or any one of the implementations of the first aspect;

The first detection module is used to input the sample image into the target character detection model to obtain the position and rotation angle of the text content in the sample image, and the target character detection model is the character according to the second aspect of the present invention The training method of the detection model is trained;

A first rotation module, configured to use the position of the text content and the rotation angle to rotate the text content in the sample image to obtain the target text content;

The second updating module is configured to update the parameters of the character recognition model based on the target text content and the text label of the sample image, so as to obtain the target character recognition model.

According to the eighth aspect, the embodiment of the present invention also provides a character recognition module, including:

The fifth acquisition module is used to acquire the image to be processed under the target scene;

The second detection module is used to input the image to be processed into the target character detection model to obtain the position and rotation angle of the text content in the image to be processed, and the target character detection model is according to the second aspect of the present invention The training method training of the character detection model obtains;

The second rotation module is used to rotate the text content in the image to be processed by using the position and rotation angle of the text content in the image to be processed to obtain the text content to be recognized;

A recognition module, configured to input the text content to be recognized into a target character recognition model to obtain a character recognition result of the image to be processed, and the target character recognition model is the character recognition model according to the third aspect of the present invention Trained by the training method.

According to a ninth aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, the memory and the processor are connected to each other in communication, the memory stores computer instructions, and the processor By executing the computer instructions, the method for generating a sample image described in the first aspect or any implementation manner of the first aspect is executed, or the method for training a character detection model described in the second aspect is executed, or, Execute the character recognition model training method described in the third aspect, or execute the character recognition method described in the fourth aspect.

According to a tenth aspect, an embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the first aspect or any of the first aspects. The method for generating a sample image described in one embodiment, or, execute the character detection model training method described in the second aspect, or execute the character recognition model training method described in the third aspect, or execute the first aspect The character recognition method described in the four aspects.

It should be noted that for the corresponding beneficial effects of the sample image generation device, character detection model training device, character recognition model training device, character recognition device, electronic equipment and computer-readable storage medium provided by the embodiments of the present invention, please refer to The method for generating the sample image, the method for training the character detection model, the method for training the character recognition model, and the method for character recognition have corresponding beneficial effect descriptions, which will not be repeated here.

Description of drawings

In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that need to be used in the specific implementation or description of the prior art. Obviously, the accompanying drawings in the following description The drawings show some implementations of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

1 is a flowchart of a method for generating a sample image according to an embodiment of the present invention;

2 is a flowchart of a method for generating a sample image according to an embodiment of the present invention;

Fig. 3 is a schematic diagram of character encoding according to an embodiment of the present invention;

4 is a schematic structural diagram of a backdiffusion process according to an embodiment of the present invention;

Fig. 5 is the flowchart of the training method of the character detection model according to the embodiment of the present invention;

6 is a schematic structural diagram of a character detection model according to an embodiment of the present invention;

Fig. 7 is the flowchart of the training method of the character recognition model according to the embodiment of the present invention;

8 is a flow chart of a character recognition method according to an embodiment of the present invention;

Fig. 9 is a structural block diagram of a device for generating a sample image according to an embodiment of the present invention;

Fig. 10 is a structural block diagram of a training device for a character detection model according to an embodiment of the present invention;

Fig. 11 is a structural block diagram of a training device for a character recognition model according to an embodiment of the present invention;

Fig. 12 is a structural block diagram of a character recognition device according to an embodiment of the present invention;

Fig. 13 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of the present invention.

As mentioned above, the printed character image generated by using a simple character style cannot be well integrated with the background image in the target scene, which has a great negative impact on character recognition. In the case of scarce sample size, related techniques generally use generative models to expand sample data sets. Among them, most of the generative models adopt the adversarial neural model, but due to the adversarial nature of the adversarial neural model itself, it is difficult to achieve a balance in training. Based on this, the sample image generation method provided by the embodiment of the present invention expands the sample data set based on the target diffusion model when the sample data set is small, and improves the fidelity of the obtained sample image on this basis . Based on the generated sample image, the embodiment of the present invention also provides a character detection model training method, which is used to train the target character detection model for detecting the position and rotation angle of the text content; further, based on the sample character image, The embodiment of the present application also provides a character recognition model training method, which is used for training to obtain a target character recognition model for character recognition.

Based on the target character recognition model obtained through training, the character recognition method provided by the embodiment of the present invention can be applied to character recognition on clothing, and can also be applied to character recognition on banners, and so on. The specific application scenarios for character recognition are set according to actual needs, and are not limited here.

According to an embodiment of the present invention, a method for generating a sample image, a method for training a character recognition model, and an embodiment of a character recognition method are provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be performed in a group of computers such as Instructions are executed in a computer system and, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

In this embodiment, a method for generating a sample image is provided, which can be used in electronic devices, such as computers, servers, etc. FIG. 1 is a flowchart of a method for generating a sample image according to an embodiment of the present invention, as shown in FIG. 1 , The process includes the following steps:

S11. Obtain target text and style features and first sample characters in the target scene.

Wherein, the target text and style features are obtained by encoding the text style of real character images in the target scene.

As mentioned above, the target scene is set according to actual needs. For example, the trained character recognition model is used to recognize characters on clothing, and accordingly, the real character image in the target scene is the real character image on clothing. The character image on the clothing is cut out from the real scene collected by the image acquisition device to obtain the real character image in the target scene.

The first sample characters are web-printed characters of different styles, or other forms of characters, etc., and there is no limitation on the form and source of the characters, which are specifically set according to actual needs.

The target text and style features in the target scene are obtained through a text style encoder. The input of the text style encoder is a character image, and the output is the text and style features of the character image. The function of the text style encoder is to extract the input Text and style features of character images, based on which style encoders are built based on feature extraction models. The feature extraction models here include but are not limited to MobileNetV2 model, ResNet or VGG, etc.

S12. Process the first sample character, the target text, and the style feature based on the target diffusion model to obtain a sample character image in the target scene.

Wherein, the target diffusion model is used to transfer the font style of the first sample character to the real character image to generate the sample character image.

The target diffusion model is used to implement font style migration. After obtaining the first sample character, target text and style features, input it into the target diffusion model to obtain the sample character image in the target scene. Specifically, the target diffusion model transfers the font style of the first sample character to the real character image to generate a sample character image. Taking the target scene as character recognition on clothing as an example, based on the target diffusion model, the font style in the first sample character is transferred to the actual personnel clothing scene, and a realistic sample character image is generated.

S13, splicing the sample character image and the background image in the target scene to obtain a sample image.

The background image in the target scene is the background of the character image, and what is obtained in the above S12 is only a sample character image, which is a character image that combines the text and style features of the target scene. In order to obtain a specific sample image, it is necessary to Blend the character image into the background of the target scene. Specifically, the sample character image is spliced with the background image in the target scene. The splicing may be placing the sample character image at any position of the background image, or placing the sample character image at any position of the background image at any angle, and so on. After stitching, a sample image including the background of the target scene and sample characters is obtained.

Here, there is no limitation on the number of generated sample images, which can be set according to actual needs.

The sample image generation method provided in this embodiment uses the target text and style features obtained from the real character image in the target scene as the conditions of the target diffusion model, and uses the target diffusion model to migrate the font style of the first sample character to the real character image, so as to ensure that the text and image in the generated sample character image can achieve a unified style and be closer to the real text style of the target scene, and then splicing the sample character image with the background image in the target scene can obtain a realistic sample character image. Using this method, a large number of realistic sample images in the target scene can be generated. On this basis, the training of the preset character recognition model can improve the accuracy of the obtained target character recognition model.

In this embodiment, a method for generating a sample image is provided, which can be used in electronic devices, such as computers, servers, etc. FIG. 2 is a flow chart of a training method for a character recognition model according to an embodiment of the present invention, as shown in FIG. 2 , the process includes the following steps:

S21. Obtain the target text and style features in the target scene and the font style of the first sample character.

Wherein, the target text and style features are obtained by encoding the text style of real character images in the target scene.

Specifically, the above S21 includes:

S211. Acquire real character images.

The real character image is obtained by extracting the character image from the image collected in the target scene, and the specific extracting method is not limited here.

S212. Perform image feature extraction on the real character image to obtain an image style code.

Image feature extraction can be realized through image feature extraction models, or through image processing. Taking the feature extraction model as an example, the real character image is input into the feature extraction model, and the output of the model is the image style code. For example, the local feature or key point feature is extracted from the input real character image to obtain the image style code .

S213, performing text feature extraction on the text content in the real character image to obtain a text code.

The text content is the text line in the real character image, and the text feature is extracted from the text line to obtain the text code. Text feature extraction can also be realized by a text feature extraction model, the text feature extraction module includes but not limited to feature extraction based on bag of words model, feature extraction based on flat-inverse document frequency (ie, TF-IDF) or word-based Vector feature extraction and so on.

S214, merging the image style coding and the text coding to obtain the target text and style features.

The fusion method of image style coding and text coding includes but not limited to concatenation of two coding vectors, or weighted sum of two coding vectors, etc., and the target text and style features are obtained after fusion processing.

In some embodiments, the above S214 includes:

(1) Attention processing is performed on image style encoding and text encoding, and the attention between image style encoding and text encoding is obtained.

(2) Integrate attention with text encoding, and process the fusion result through a feed-forward network to obtain target text and style features.

Attention processing is implemented based on a multi-head attention module. The input is image style encoding and text encoding, and the output is the attention between image style encoding and text encoding, so that different text characters focus on different parts of a given style sample. After the attention between the two is obtained, the attention is fused with the text encoding to obtain the fusion result, and then the fusion result is passed through the feedforward network to obtain the target text and style features. Among them, the fusion of attention and text encoding can be stacked in the channel direction of multi-dimensional matrix. The role of the feed-forward network is to integrate the image features of the character region and the semantic features of the character content in the region.

The fusion of image style coding and text coding is achieved through attention, so that different text content focuses on different parts of a given image style coding, which improves the reliability and authenticity of the fusion results obtained. On this basis, feedforward The processing of the network can extract richer semantic features, which further ensures the authenticity of the real character image text style encoding, so that when the target text and style features are used as the input conditions of the target diffusion model, the target diffusion model can be diffused and obtained more effectively. Real sample images.

As a specific application example, take character recognition on clothing as an example. Fig. 3 shows a specific application example of text style coding. Specifically, as shown in Figure 3, the above method of determining the target text and style features includes: extracting the character image on the clothing from the real scene captured by the camera, inputting it into the MobileNetV2 model pre-trained on Imagenet, discarding the top full The connection layer uses the average pooling layer to extract local features as image style encoding. Since the MobileNetV2 model here is essentially a classification model, the fully connected layer is connected at the end of the model to implement the classification function. In this embodiment, the model is only needed as a feature extractor, so the fully connected layer is removed; that is, the network structure before the fully connected layer of the classification module is the feature extraction module, and the purpose of adding average pooling is to to filter. Of course, in addition to the average pooling, the maximum pooling layer and the like can also be used, which is not limited here. Further, use the embedding layer to encode the current text line according to the text content in the character image, and encode according to the character sequence of English and numbers to represent the text encoding. The extracted image style encoding and text encoding are input to a multi-head attention layer, and by computing the attention between the text encoding and the extracted features, different text characters focus on different parts of a given style sample. Finally, the output of the multi-head attention mechanism is added to the text encoding, and then the final target text and style feature output are obtained through the feedforward network.

S215. Acquire a first sample character.

S22. Process the first sample character, the target text, and the style feature based on the target diffusion model to obtain a sample character image in the target scene.

Wherein, the target diffusion model is used to transfer the font style of the first sample character to the real character image to generate the sample character image.

As mentioned above, the target diffusion model is used to implement font style transfer, and its output is a sample character image.

In some embodiments, the training method of target diffusion model includes:

(1) Obtain the text and style features of the second sample character and the sample image in the target scene.

(2) During the forward diffusion process of the preset diffusion model, a noise image is generated based on the second sample character.

(3) Based on the noise image and the text and style features of the sample image in the target scene, the reverse diffusion process of the preset diffusion model is trained to determine the target diffusion model.

The second sample character is a randomly generated image, for example, using an opencv tool to generate a second sample character of random length, random color, and random size as the original image. The acquisition method of the text and style features of the sample image in the target scene is similar to the acquisition method of the target text and style features described above, and will not be repeated here.

For the preset diffusion model, it includes the forward diffusion process and the reverse diffusion process, wherein the forward diffusion process is used to change the distribution of the original image into a standard Gaussian distribution, and the reverse diffusion process is used to Generate character images in the target scene. Taking character recognition on clothing as an example, noise is randomly added to the original image during forward diffusion. For the noise mechanism β ₁ ,…β _T , use β _t =0.02+Exponential(1×10 ^-5 ,0.4), where Exponential(1×10 ^-5 ,0.4) means log(1×10 ^-5 ) to log( 0.4) between floating point numbers. After T-step iterations, the distribution of the original image is finally transformed into a standard Gaussian distribution. Wherein, β _i represents the noise added in the i-th iteration (i=1, 2, .

The back-diffusion process is implemented based on the UNet model as shown in Figure 4. As shown in Figure 4, the UNet model consists of a downsampling block and an upsampling block, and uses long-range convolutional skip connections. It mainly uses two Types of blocks, convolutional blocks and attentional blocks. The convolutional block consists of 3 convolutional layers and a convolutional skip connection and applies a conditional affine transformation to the output of each convolutional layer, the scale and bias of each conditional affine transformation is parameterized by the output of the fully connected layer. The attention block consists of 2 multi-head attention layers and a feed-forward network. The first attention layer performs attention between the text sequence latent variables of the input noisy image and the text and style features of the sample image, while the second attention layer performs self-attention. Conditional affine transformations are performed after each attention layer and feed-forward network using layer normalization. As shown in Figure 4, the input of the attention block consists of two parts, one is the original image used as supervision, and the other is the text and style features of the sample image under the target scene, that is, the noise image is used as the supervision, and the real clothing scene The text and style features of the sample images below are used as the reverse process of training the diffusion model.

In the reverse process, the noisy image output by a forward process is sampled from a spherical Gaussian function to obtain a noisy image with a long Markov chain, and the conditional latent variables containing the semantics of clothing characters (i.e., the text of the sample image and Style feature) denoises the noisy image in the long Markov chain to generate the character image of the clothing scene style.

The second sample character is used as supervision, and the text and style features of the sample image in the target scene are used as conditions to train the reverse diffusion process of the preset diffusion model, which can ensure that the obtained target diffusion model can generate an image that fits the target scene .

S23. Concatenate the sample character image with the background image in the target scene to obtain a sample image.

Specifically, the above S23 includes:

S231. Acquire a background image in the target scene, and identify a region of interest on the background image to obtain an image of the region of interest.

When the sample character image is stitched into the background image, it is generally stitched in the region of interest in the background image. For example, the characters on clothing are generally the upper body of the person. Based on this, it is necessary to recognize the upper body of the person in the collected background image. At this time, the half body of the person is the region of interest in the background image.

Identify the region of interest on the background image, determine the location of the region of interest, and then use the location of the region of interest to extract a partial image from the background image to obtain an image of the region of interest.

S232. Rotate the sample character image at any angle to obtain a rotated sample character image.

The rotation angle of the sample character image is set according to actual requirements. After the sample character image is rotated by any angle, the rotated sample character image is obtained. In order to facilitate the recording of the rotation angle, the rotation angle is specified as the acute angle between the short side of the rotation target frame and the positive direction of the x-axis, where the counterclockwise direction is specified as a positive angle, and the clockwise direction is a negative angle. Therefore, the angle range is [ -90,90).

S233, splicing the rotated sample character image and the ROI image to obtain a sample image.

The splicing method is to paste the rotated sample character image on the image of the region of interest to obtain the sample image.

Further, in order to obtain a large number of sample images, a large number of first sample characters with different styles can be used as the input of the target diffusion model, and real scene images of different styles in the target scene can be collected to obtain the target text and style features of the real scene image, Then, the target text and style features are used as the conditions of the target diffusion model to generate a large number of sample character images, thereby generating a large number of sample images.

The sample image generation method provided in this embodiment extracts image features and text features respectively for real character images, and then fuses the two, so that the obtained target text and style features include both image features and text features. Text features, which improve the reliability of the resulting target text and style features. Through the identification of the region of interest, it is convenient to stitch the sample character image to the region of interest, so as to be close to the use in the target scene; at the same time, the sample character image is rotated at any angle, so that a large number of sample images can be generated, thereby enriching the sample the number of images.

In this embodiment, a method for training a character detection model is provided, which can be used in electronic equipment, such as computers, servers, etc. FIG. 5 is a flowchart of a method for training a character detection model according to an embodiment of the present invention, as shown in FIG. 5 The process includes the following steps:

S31. Acquire a sample image.

Wherein, the sample image is obtained according to the method for generating a sample image described in any one of the above. For the generation of the sample image, please refer to the above description, and details will not be repeated here.

S32. Acquire tag data of the sample image.

Wherein, the label data includes the position information of the text content in the sample image and the target rotation angle.

The position information of the text content and the target rotation angle are recorded when the sample character image is stitched together with the background image in the target scene.

S33. Input the sample image into the character detection model to obtain predicted position information and predicted rotation angle of the text content in the sample image.

The input of the character detection model is a sample image, and the output is the predicted position information and predicted rotation angle of the text line in the sample image. Among them, the predicted position information can be represented by the center point, length and width, or can be represented by the coordinates of the upper left corner and the lower right corner of the prediction box where the text line is located, etc., and there is no specific limitation on it. .

S34. Based on the predicted position information, predicted rotation angle and label data, update the parameters of the character detection model to determine the target character detection model.

As a specific application example of the character detection model in this embodiment, the character detection model is built with resnet-18 as the backbone network, and then four output feature map branches are connected, which are target center point heat map and sampling offset The amount, the size of the target frame and the rotation angle of the target frame are trained to predict the specific position of the target by training these four branches. Specifically, the obtained sample image is scaled to a resolution of 3*320*192, and used as the input of the character detection model, which outputs a center point heat map, an offset feature map, a target frame size feature map, and an angle feature map. Combined with the label of the sample image, the loss value between the label and the output of the character detection model is continuously iterated, and parametric training is performed to update the parameters of the character detection model, and finally determine the target character detection unit.

Specifically, for the size of the feature map output by the backbone network, the Gaussian kernel is used to calculate the real value of the heat map. When the position of the enumeration block and the coordinates of the real central key point are close to coincident, the Gaussian kernel output value is close to 1; when the enumeration The Gaussian kernel output value is close to 0 when the block position is very different from the true central keypoint. The output size of the center point heat map is a quarter of the size of the original image. Since the detection target only has one category of character boxes, the output heat map size is 80*48*1. During the training process, the center point heatmap is trained with focal loss. In the inference process, 3×3 maximum pooling is performed on the predicted center point heat map to calculate the center point that meets the detection threshold.

Since the image has been down-sampled by R=4, the offset of the x-axis and y-axis of the center point is calculated according to the downsampling rate of the feature map, and the loss function about the offset is set so that the trained network can compensate for the offset of the center point Value, correct the position of the detection frame. The output feature map size of this branch is 80*48*2, which is used to predict the offset of the x-axis and y-axis. This offset value is trained with L1 loss.

The size of the target box size feature map output by the character detection model is 80*48*2, and the two channels correspond to the prediction of height and width respectively. The predicted height and width are mapped to the size of the original image, and the L1 loss is used to train this branch, and then The height and width predicted by the character detection model gradually converge to the character frame of the original image.

The character detection model adds a detection head of angle information after the feature extractor of the backbone network. The size of this feature map is 80*48*1, which is used to predict the rotation angle of the target frame and feed back the target angle error to the character detection model. This enables the character detection model to learn the angle information of the target.

The character detection model training method provided in this embodiment, on the basis of a large number of realistic sample images obtained in the target scene, performs character detection model training, which can improve the accuracy of the obtained target character detection model; Moreover, the trained target character detection model also outputs the rotation angle of the text content, which improves the detection accuracy of the target character detection model. Moreover, during subsequent character recognition, the detected text content can be rotated by using the rotation angle, so as to further improve the accuracy of character recognition.

In this embodiment, a method for training a character recognition model is provided, which can be used in electronic devices, such as computers, servers, etc. FIG. 6 is a flow chart of a method for training a character recognition model according to an embodiment of the present invention, as shown in FIG. 6 The process includes the following steps:

S41. Acquire a sample image.

Wherein, the sample image is obtained according to the method for generating a sample image described in any one of the above. For the generation of the sample image, please refer to the above description, and details will not be repeated here.

S42. Input the sample image into the target character detection model to obtain the position and rotation angle of the text content in the sample image.

Wherein, the target character detection model is trained according to the above-mentioned character detection model training method. For the specific details of the target character detection model, please refer to the above description, which will not be repeated here.

S43, using the position and rotation angle of the text content to rotate the text content in the sample image to obtain the target text content.

The way to rotate the text content can be to use the position of the text content to extract the text line first, and then combine the rotation angle to rotate the text line to obtain the target text content; it is also possible to use the position of the text content and the rotation angle to correct the keystone of the text content, specifically Specifically, the perspective transformation matrix is calculated by using the position and the rotation angle, and then the perspective transformation matrix is used to perform perspective transformation on the text content in the sample image to obtain the target text content.

Of course, the target text content can also be obtained by rotating the text content in other ways based on the position and rotation angle of the text content, which is not limited here.

S44. Based on the target text content and the text label of the sample image, update the parameters of the character recognition model to obtain the target character recognition model.

The character recognition model can be a pre-trained character recognition model, for example, it can be a pre-trained English character and number character classifier for recognizing English and numbers. During the training process, the parameters of the character recognition model are adjusted to fix the parameters of the character recognition model. The parameter adjustment process can be to scale the generated sample image to a resolution of 3*32*384 according to the width and height ratio of the original image. If the width of the original image is less than 384 after scaling, the three channels will be filled with the average value of the three channels respectively. If the width is greater than 384 , the images are fed into the pre-trained character recognition model in batches with 384 width truncated. The character recognition model is built with resnet18 as the backbone network, and classifies 52 English uppercase and lowercase letters and 10 numbers through the softmax activation function. Train CTCloss using text classification, and set blank for repeated characters input. As a category, the final classification feature map is 1*48*63, CTCloss calculation is performed on this feature map, and the network parameters are continuously adjusted through the Adam optimizer to fit the correct string, thereby fixing the parameters of the character recognition model, and obtaining Pretrained character recognition model. Further, apply the pre-trained character recognition model to the target scene, use the target character detection model to detect the sample image, and rotate the text content based on the position and rotation angle of the text content to obtain the target text content, use the pre-trained character recognition The model calculates the loss of the target text content and the sample image text label, and updates the parameters of the pre-trained character recognition model based on the loss to obtain the target character recognition model, making the target character recognition model more suitable for character/text recognition in the target scene.

The character recognition model training method provided in this embodiment uses the predicted position and rotation angle to perform rotation correction on the text content before inputting the detected text line into the character recognition model, so as to ensure the accuracy of the obtained target text content. The angle is unified, and then the parameters of the character recognition model are updated by using the target text content with the same angle, which further improves the accuracy of the trained target character recognition model.

In this embodiment, a character recognition method is provided, which can be used in electronic devices, such as computers, servers, mobile terminals, etc. FIG. 8 is a flowchart of a character recognition method according to an embodiment of the present invention. As shown in FIG. 8, the The process includes the following steps:

S51. Acquire the image to be processed under the target scene.

The image to be processed is an image in the target scene, which may be collected by a collection device in the target scene, or stored in an electronic device, etc., and its source is not limited here.

S52. Input the image to be processed into the target character detection model to obtain the position and rotation angle of the text content in the image to be processed.

Wherein, the target character detection model is obtained through training according to the above-mentioned character detection model training method. For the specific structural details of the target character detection model, please refer to the above description, which will not be repeated here.

S53, using the position and rotation angle of the text content in the image to be processed to rotate the text content in the image to be processed to obtain the text content to be recognized.

The specific rotation processing is similar to the rotation processing in S43 above, and will not be repeated here.

S54, input the text content to be recognized into the target character recognition model, and obtain the character recognition result of the image to be processed.

Wherein, the target character recognition model is obtained through training according to the above-mentioned character recognition model training method.

In some implementations, if the trained target character recognition model can only recognize English characters and numbers, for the target character recognition model, Chinese characters may be detected, resulting in misclassification of the target character recognition model, that is, the output Garbled characters mixed with long English and numbers. Based on this, the ratio threshold of the number of Chinese and English occurrences is set. Since the prediction scores of Chinese characters are generally low, when the ratio of the number of characters below the classification score threshold to the length of the string is greater than the set ratio threshold, it is considered The character string is in Chinese and will not be output. In other cases, the character recognition result will be output.

Since the rotation correction depends on the output of the target character detection model rather than independent rotation detection, based on this, the text content is rotated to obtain the corrected text content to be recognized, and then character recognition is performed on this basis, which improves the accuracy of the character. Accuracy of recognition results.

The character recognition method provided in this embodiment, because the target character detection model and the target character recognition model are trained with a large number of realistic sample images, has higher character detection and character recognition accuracy, and the target character detection model and the target character recognition model are used The recognition model performs character recognition on the image to be processed, and can obtain more accurate character recognition results.

In this embodiment, a device for generating a sample image, a device for training a character detection model, a device for training a character recognition model, and a device for character recognition are also provided. The description will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

This embodiment provides a device for generating a sample image, as shown in FIG. 9 , including:

The first obtaining module 61 is used to obtain the target text and style feature and the first sample character in the target scene, and the target text and style feature are obtained by performing text style coding on the real character image in the target scene;

A style processing module 62, configured to process the first sample character, the target text and style features based on a target diffusion model to obtain a sample character image in the target scene, and the target diffusion model is used to convert the The font style of the first sample character is transferred to the real character image to generate the sample character image;

The splicing module 63 is configured to splice the sample character image with the background image in the target scene to obtain a sample image.

In some implementations, the first acquisition module 61 includes:

a first acquiring unit, configured to acquire the real character image;

The first feature extraction unit is used to perform image feature extraction on the real character image to obtain an image style code;

The second feature extraction unit is used to perform text feature extraction on the text content in the real character image to obtain a text code;

A fusion unit, configured to fuse the image style code and the text code to obtain the target text and style features.

In some embodiments, the fusion unit comprises:

An attention processing subunit, configured to perform attention processing on the image style code and the text code, to obtain the attention between the image style code and the text code;

The fusion subunit is used to fuse the attention with the text code, and process the fusion result through a feed-forward network to obtain the target text and style features.

In some embodiments, the splicing module 63 includes:

The second acquisition unit is configured to acquire a background image in the target scene, and identify a region of interest on the background image to obtain an image of the region of interest;

The first rotation unit is used to rotate the sample character image at any angle to obtain the rotated sample character image;

A splicing unit, configured to splice the rotated sample character image and the ROI image to obtain the sample image.

In some embodiments, the training module of the target diffusion model includes:

A fourth acquisition unit, configured to acquire the second sample character and the text and style features of the sample image in the target scene;

A generating unit, configured to generate a noise image based on the second sample character during the forward diffusion process of the preset diffusion model;

A training unit, configured to train a reverse diffusion process of the preset diffusion model based on the noise image and the text and style features of the sample image in the target scene, so as to determine the target diffusion model.

This embodiment provides a training device for a character detection model, as shown in Figure 10, including:

The second acquisition module 71 is configured to acquire a sample image, the sample image is obtained according to the method for generating a sample image described in the first aspect of the present invention or any one of the implementations of the first aspect;

The third obtaining module 72 is used to obtain the tag data of the sample image, the tag data includes the position information of the text content in the sample image and the target rotation angle;

The first prediction module 73 is used to input the sample image into the character detection model to obtain predicted position information and predicted rotation angle of the text content in the sample image;

The first updating module 74 is configured to update the parameters of the character detection model based on the predicted position information, the predicted rotation angle and the label data, so as to obtain a target character detection model.

This embodiment provides a training device for a character recognition model, as shown in FIG. 11 , including:

The fourth acquisition module 81 is configured to acquire a sample image, the sample image is obtained according to the method for generating a sample image described in the first aspect of the present invention or any one of the implementations of the first aspect;

The first detection module 82 is used to input the sample image into the target character detection model to obtain the position and rotation angle of the text content in the sample image, and the target character detection model is according to the second aspect of the present invention The training method of the character detection model is trained;

The first rotation module 83 is configured to use the position of the text content and the rotation angle to rotate the text content in the sample image to obtain the target text content;

The second update module 84 is configured to update the parameters of the character recognition model based on the target text content and the text label of the sample image, so as to obtain the target character recognition model.

This embodiment provides a character recognition device, as shown in Figure 12, including:

The fifth acquisition module 91 is used to acquire the image to be processed under the target scene;

The second detection module 92 is configured to input the image to be processed into a target character detection model to obtain the position and rotation angle of the text content in the image to be processed, and the target character detection model is developed according to the second aspect of the present invention Obtained by the training method training of the character detection model described above;

The second rotation module 93 is configured to use the position and rotation angle of the text content in the image to be processed to rotate the text content in the image to be processed to obtain the text content to be recognized;

A recognition module 94, configured to input the text content to be recognized into a target character recognition model to obtain a character recognition result of the image to be processed, and the target character recognition model is the character recognition model according to the third aspect of the present invention trained by the training method.

The character recognition model training device and the character recognition device in this embodiment are presented in the form of functional units, where a unit refers to an ASIC circuit, a processor and a memory that execute one or more software or fixed programs, and/or Other devices that can provide the above functions.

Further functional descriptions of the above-mentioned modules are the same as those in the above-mentioned corresponding embodiments, and will not be repeated here.

The embodiment of the present invention also provides an electronic device, which has the above-mentioned sample image generation device shown in FIG. 9, or the character detection model training device shown in FIG. 10, or the character recognition model training device shown in FIG. 11, Or the character recognition device shown in Figure 12.

Please refer to FIG. 13. FIG. 13 is a schematic structural diagram of a terminal provided in an optional embodiment of the present invention. As shown in FIG. 13, the terminal may include: at least one processor 101, such as a CPU (Central Processing Unit, central processing unit ), at least one communication interface 103, memory 104, at least one communication bus 102. Wherein, the communication bus 102 is used to realize connection and communication between these components. Wherein, the communication interface 103 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 103 may also include a standard wired interface and a wireless interface. The memory 104 may be a high-speed RAM memory (Random Access Memory, volatile random access memory), or a non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory 104 may also be at least one storage device located away from the aforementioned processor 101 . Wherein the processor 101 can be combined with the device described in FIG. 9 or FIG. 10 or FIG. 11 or FIG. 12, the application program is stored in the memory 104, and the processor 101 calls the program code stored in the memory 104 to perform any of the above methods step.

Wherein, the communication bus 102 may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (EISA for short) bus or the like. The communication bus 102 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in FIG. 13 , but it does not mean that there is only one bus or one type of bus.

Wherein, the memory 104 may include a volatile memory (English: volatile memory), such as a random-access memory (English: random-access memory, abbreviated as RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory), hard disk (English: hard diskdrive, abbreviated: HDD) or solid-state hard disk (English: solid-state drive, abbreviated: SSD); the memory 104 can also include the above-mentioned types of memory The combination.

Wherein, the processor 101 may be a central processing unit (English: central processing unit, abbreviated: CPU), a network processor (English: network processor, abbreviated: NP) or a combination of CPU and NP.

Wherein, the processor 101 may further include a hardware chip. The aforementioned hardware chip may be an application-specific integrated circuit (English: application-specific integrated circuit, abbreviation: ASIC), a programmable logic device (English: programmable logic device, abbreviation: PLD) or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (English: complex programmable logic device, abbreviated: CPLD), field-programmable logic gate array (English: field-programmable gate array, abbreviated: FPGA), general array logic (English: generic array logic , Abbreviation: GAL) or any combination thereof.

Optionally, the memory 104 is also used to store program instructions. The processor 101 can invoke program instructions to implement the method for generating a sample image, the method for training a character detection model, the method for training a character recognition model, or the method for character recognition as shown in any embodiment of the present application.

An embodiment of the present invention also provides a non-transitory computer storage medium, the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the sample image generation method and character detection method in any of the above method embodiments The training method of the model, the training method of the character recognition model, or the character recognition method. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a flash memory (Flash Memory), a hard disk (Hard Disk) Disk Drive, abbreviation: HDD) or solid-state drive (Solid-State Drive, SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memory.

Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention. within the bounds of the requirements.

Claims (10)

1. A method for generating a sample image, comprising: Acquiring target text and style features and first sample characters in the target scene, where the target text and style features are obtained by text style encoding of real character images in the target scene; Process the first sample character, the target text and style features based on a target diffusion model to obtain a sample character image in the target scene, and the target diffusion model is used to convert the first sample character The font style is transferred to the real character image to generate the sample character image; The sample character image is spliced with the background image in the target scene to obtain a sample image. 2. method according to claim 1, is characterized in that, described acquisition target text and style feature under target scene, comprise: Acquiring the real character image; Carry out image feature extraction to described real character image, obtain image style coding; Carrying out text feature extraction to the text content in described real character image, obtains text coding; The image style coding and the text coding are fused to obtain the target text and style features. 3. The method according to claim 2, wherein said fusion of said image style code and said text code to obtain said target text and style feature comprises: performing attention processing on the image style code and the text code to obtain the attention between the image style code and the text code; The attention is fused with the text code, and the fusion result is processed by a feed-forward network to obtain the target text and style features. 4. The method according to claim 1, wherein the splicing the sample character image with the background image under the target scene to obtain a sample image comprises: Acquiring a background image in the target scene, and identifying a region of interest on the background image to obtain an image of a region of interest; Rotating the sample character image at any angle to obtain a rotated sample character image; The rotated sample character image is spliced with the ROI image to obtain the sample image. 5. The method according to any one of claims 1-4, wherein the training method of the target diffusion model comprises: Acquiring the text and style features of the second sample character and the sample image in the target scene; During the forward diffusion process of the preset diffusion model, a noise image is generated based on the second sample character; Based on the noise image and the text and style features of the sample image in the target scene, the reverse diffusion process of the preset diffusion model is trained to determine the target diffusion model. 6. A training method for a character detection model, comprising: Obtaining a sample image, the sample image is obtained according to the method for generating a sample image according to any one of claims 1-5; Acquiring label data of the sample image, the label data including the position information of the text content in the sample image and the target rotation angle; Inputting the sample image into the character detection model to obtain predicted position information and predicted rotation angle of the text content in the sample image; Based on the predicted position information, the predicted rotation angle and the label data, the parameters of the character detection model are updated to obtain a target character detection model. 7. A training method for a character recognition model, comprising: Obtaining a sample image, the sample image is obtained according to the method for generating a sample image according to any one of claims 1-5; The sample image is input into the target character detection model to obtain the position and rotation angle of the text content in the sample image, and the target character detection model is obtained according to the training method of the character detection model described in claim 6; using the position of the text content and the rotation angle to rotate the text content in the sample image to obtain the target text content; Based on the target text content and the text label of the sample image, the parameters of the character recognition model are updated to obtain the target character recognition model. 8. A character recognition method, characterized in that, comprising: Obtain the image to be processed under the target scene; Input the image to be processed into the target character detection model to obtain the position and rotation angle of the text content in the image to be processed, and the target character detection model is obtained according to the training method of the character detection model described in claim 6 of; Using the position and rotation angle of the text content in the image to be processed to rotate the text content in the image to be processed to obtain the text content to be recognized; Inputting the text content to be recognized into the target character recognition model to obtain the character recognition result of the image to be processed, the target character recognition model is obtained by training according to the character recognition model training method described in claim 7 . 9. An electronic device, characterized in that it comprises: A memory and a processor, the memory and the processor are connected in communication with each other, computer instructions are stored in the memory, and the processor performs any one of claims 1-5 by executing the computer instructions The generation method of described sample image, or, carry out the training method of the character detection model described in claim 6, perhaps, carry out the training method of the character recognition model described in claim 7, perhaps, carry out the training method described in claim 8 character recognition method. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to make a computer perform the processing of the sample image according to any one of claims 1-5. The generating method, or, executes the character detection model training method described in claim 6, or executes the character recognition model training method described in claim 7, or executes the character recognition method described in claim 8.

Images