CN114996515A - Training method of video feature extraction model, text generation method and device - Google Patents

Abstract

The present disclosure relates to a training method, text generation method and device for a video feature extraction model, belonging to the technical field of computers. In the embodiment of the present disclosure, the image information and text information of the sample video and the text label and image label of the sample video are used to perform model training on the video feature extraction model, and a model training method based on dual training tasks is provided. When the text generation task is the main task and the image reconstruction task is the auxiliary task, since the image label of the sample video represents the image reconstruction feature, in the process of model training, the video feature extraction model can improve the image feature extraction ability. , and then obtain high-quality image features. On the basis of obtaining high-quality image features, a video feature extraction model with better text generation capability can be trained, which improves the training effect of the video feature extraction model.

Description

Training method, text generation method and device for video feature extraction model

technical field

The present disclosure relates to the field of computer technologies, and in particular, to a method for training a video feature extraction model, a method for generating text, and an apparatus.

Background technique

With the rapid development of computer technology and Internet technology, video processing technology has gradually become an emerging research hotspot. In the video processing technology, it is usually necessary to extract video features that can characterize video content, and then use the video features to perform processing processes such as video recommendation, video classification, or video search.

At present, before feature extraction is performed on a video, a video classification model is usually trained according to the image information of multiple sample videos and the category labels of the multiple sample videos, and then the images in the video are classified by the trained video classification model. process to obtain the category features of the video. However, the above-mentioned video classification model has weak feature extraction capability, which is not conducive to the subsequent processing of video recommendation, video classification or video search.

SUMMARY OF THE INVENTION

The present disclosure provides a video feature extraction model training method, text generation method and device, which can train a video feature extraction model with better text generation capability and improve the training effect of the video feature extraction model. The technical solutions of the present disclosure are as follows:

According to a first aspect of the embodiments of the present disclosure, a training method for a video feature extraction model is provided, the method comprising:

Obtain the image information, text information, image label and text label of the sample video, the image label represents the image reconstruction feature, and the text label represents the content description text of the sample video;

Input the image information and the text information into the video feature extraction model, perform feature extraction on the image information through the image feature extraction sub-model of the video feature extraction model, obtain the image features of the sample video, and extract the features of the model through the video feature extraction model. The embedding layer of the fusion sub-model processes the text information to obtain the text features of the sample video, and the image features and the text features are feature-fused through the feature fusion layer of the feature fusion sub-model to obtain the fusion features of the sample video. ;

Image restoration is performed on the image features in the fusion feature through the image reconstruction sub-model of the video feature extraction model to obtain an image training result of the original image size, and the fusion feature is processed through the text generation sub-model of the video feature extraction model, Get text training results;

Based on the image training result, the text training result and the image label and text label of the sample video, the model parameters of the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text generation sub-model are adjusted, to train the video feature extraction model.

In the embodiment of the present disclosure, the image information and text information of the sample video and the text label and image label of the sample video are used to perform model training on the video feature extraction model. The model can accurately extract the image features of the sample video. By constructing a feature fusion sub-model in the video feature extraction model, not only can the text features of the sample video be obtained, but also the image features and text features of the sample video. Fusion, so that based on the fusion features, on the one hand, the image reconstruction of the sample video can be performed to obtain high-quality image features, and on the other hand, the content description text of the sample video can also be generated. The model training method for training tasks, in the case where the text generation task is the main task and the image reconstruction task is the auxiliary task, since the image label of the sample video represents the image reconstruction feature, in the process of model training, the video features can be improved. The extraction model is based on the extraction ability of image features, and then obtains high-quality image features. On the basis of obtaining high-quality image features, a video feature extraction model with better text generation ability can be trained, which improves the video features. Extract the training effect of the model.

In some embodiments, the acquisition process of the image information of the sample video includes at least one of the following:

Obtain the cover image of the sample video; or, obtain at least one frame of image in the sample video.

In the embodiment of the present disclosure, by obtaining the cover image of the sample video or the image frame included in the sample video, the image information of the sample video can be quickly obtained, which not only ensures the efficiency of obtaining image information, but also enriches the types of image information. , which improves the flexibility of acquiring image information.

In some embodiments, the process of acquiring the text information of the sample video includes at least one of the following:

Obtain the description information of the sample video; obtain the title information of the sample video; obtain the subtitle information of the sample video; Identify the obtained result; obtain the audio recognition result of the sample video, where the audio recognition result is the result obtained by performing audio recognition on the background audio of the sample video.

In the embodiment of the present disclosure, by obtaining the description information, title information, subtitle information, text recognition result or audio recognition result of the sample video, the text information of the sample video can be quickly obtained. The types of text information are enriched, and the flexibility of obtaining text information is improved.

In some embodiments, the content description text is at least one of content category description text, content form description text, content topic description text, and content detail description text.

In the embodiment of the present disclosure, by setting multiple types of content description texts, on the one hand, more expressive content description texts can be generated; The content is described, which enriches the type of the generated content description text, and can more fully and completely characterize the video.

In some embodiments, feature fusion is performed on the image feature and the text feature through the feature fusion layer of the feature fusion sub-model to obtain the fusion feature of the sample video including any of the following:

Through the self-attention layer included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video;

Through the deep belief network included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video.

In the embodiment of the present disclosure, by setting a self-attention layer or a deep belief network in the feature fusion sub-model, and then using the self-attention mechanism or the deep belief network to perform feature fusion, it is possible to obtain features with more video representation capabilities, and improve the performance of the video. The accuracy of feature fusion.

In some embodiments, the fusion feature is processed by the text generation sub-model of the video feature extraction model, and the obtained text training result includes:

Through the self-attention layer included in the text generation sub-model, the fusion feature is processed to obtain the text training result.

In the embodiment of the present disclosure, by setting a self-attention layer in the text generation sub-model, and then using the self-attention mechanism to generate content description text, the accuracy of text generation is improved.

In some embodiments, the text training result includes multiple types of content description text;

The fusion feature is processed by the text generation sub-model of the video feature extraction model to obtain the text training result, the method further includes:

On the fusion feature, add the type identifier of each type;

The fusion feature is processed by the text generation sub-model of the video feature extraction model, and the text training result obtained includes:

The fusion feature after adding the type identifier is input into the text generation sub-model, and the fusion feature is processed based on the processing mechanism corresponding to each type identifier through the text generation sub-model to obtain the multiple types of content description texts.

In the embodiment of the present disclosure, by adding various types of type identifiers to the fusion feature, the text generation sub-model in the video feature extraction model can trigger the generation of multiple types of content description texts of the sample video based on the various types of type identifiers. , to ensure smooth text generation.

In some embodiments, the number of the text information is multiple;

The text information is processed by the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain the text feature of the sample video, the method further includes:

Splicing a plurality of the text information to obtain the spliced text information;

Based on the spliced text information, the step of processing the text information through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain text features of the sample video is performed.

In the embodiment of the present disclosure, when the number of text information is multiple, the spliced text information is obtained by splicing the text information, and then the spliced text information is used to perform the process of extracting text features. Various types of text information improve the accuracy of extracting text features.

In some embodiments, after splicing a plurality of the text information to obtain the spliced text information, the method further includes:

From the spliced text information, extract the characters of the previous target number;

Based on the extracted characters, the step of processing the text information through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain text features of the sample video is performed.

In the embodiment of the present disclosure, in the spliced text information, the characters of the previous target number are extracted, so as to perform the subsequent process of extracting text features based on the extracted certain number of characters, and on the basis of ensuring that sufficient text information is input , which reduces the computational complexity of the video feature extraction model and improves the efficiency of text feature extraction.

In some embodiments, the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text generation sub-model are adjusted based on the image training result, the text training result and the image labels and text labels of the sample video The model parameters of the sub-model to train this video feature extraction model include:

In the ith iteration process of model training, the model loss value of the ith iteration process is determined based on the image training results and text training results of the ith iteration process and the image labels and text labels of the sample video. i is a positive integer greater than 1;

Based on the model loss value of the i-th iterative process, adjust the model parameters of the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text generation sub-model determined in the i-1-th iteration process , and repeat the iterative process of the above training until the training meets the target condition.

In the embodiment of the present disclosure, in any iterative process of model training, the model loss value of this iterative process is used to adjust the model parameters of each sub-model in the video feature extraction model, so as to improve the performance of the video feature extraction model. Text generation ability, so as to train a video feature extraction model with high text generation ability.

In some embodiments, determining the model loss value of the i-th iteration process based on the image training results and text training results of the i-th iteration process and the image labels and text labels of the sample video includes:

Based on the image training result of the ith iteration process and the image label of the sample video, the image reconstruction loss value of the ith iteration process is determined, and the image reconstruction loss value represents the difference between the image training result and the image label ;

Based on the text training result of the ith iteration process and the text label of the sample video, determine the text generation loss value of the ith iteration process, where the text generation loss value represents the difference between the text training result and the text label ;

Based on the image reconstruction loss value and the text generation loss value, the model loss value of the i-th iteration process is determined.

In some embodiments, based on the image reconstruction loss value and the text generation loss value, determining the model loss value of the i-th iteration process includes:

Based on the image reconstruction loss value, the weight coefficient corresponding to the image reconstruction loss value, the text generation loss value, and the weight coefficient corresponding to the text generation loss value, a weighted sum is performed to obtain the model loss value of the i-th iteration process.

In the embodiment of the present disclosure, for the image reconstruction task and the text generation task of the video feature extraction model, weight coefficients corresponding to each task are respectively set, and then the loss value of each task and the weight coefficient corresponding to each task are used to determine the model loss value , which improves the accuracy of determining the loss value of the model.

In some embodiments, the text training result includes multiple types of content description text;

Based on the text training result of the ith iterative process and the text label of the sample video, determining the text generation loss value of the ith iterative process includes:

For any type, determine the loss value of the i-th iteration process on the type based on the text training result of the i-th iteration process on the type and the text label of the sample video on the type;

Based on the loss values of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, weighted summation is performed to obtain the text generation loss value of the i-th iteration process.

In the embodiment of the present disclosure, for each type involved in text generation, a weight coefficient corresponding to each type is set respectively, and then the loss value corresponding to each type and the weight coefficient corresponding to each type are used to determine the text generation loss value, which improves the Determines the accuracy of the text generation loss value.

In some embodiments, based on the loss values of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, a weighted sum is performed to obtain the i-th iteration process. Before text generation loss value, the method also includes:

For any type, based on the correct number of texts on the type and the total number of texts in the ith iteration process, determine the correct scale of the ith iteration process on the type, the correct scale representing the ith iteration process The ratio of the number of correct texts to the total number of texts in the process;

Based on the correct scale of the i-th iterative process on the type, the weight coefficient of the video feature extraction network on the type is determined, wherein the correct scale is negatively correlated with the weight coefficient.

In the embodiment of the present disclosure, for each type involved in text generation, the weight coefficient corresponding to each type is determined according to the correct ratio corresponding to each type, because the correct ratio represents the ratio of the correct text quantity to the total text quantity, and since the correct ratio represents the ratio of the correct text quantity to the total text quantity There is a negative correlation between the correct scale and the weight coefficient. Therefore, in the case of calculating the text generation loss value, set a smaller weight coefficient for the type with a large correct scale, and set a larger weight coefficient for the type with a small correct scale. In order to determine the accuracy of the weight coefficient, the accuracy of determining the text generation loss value is also improved.

According to a second aspect of the embodiments of the present disclosure, there is provided a text generation method based on a video feature extraction model, where the video feature extraction model is obtained by training based on the first aspect or the training method shown in any embodiment of the first aspect. , the method includes:

Obtain the image information and text information of the target video;

The image information and the text information are input into the video feature extraction model, and feature extraction is performed on the image information through the image feature extraction sub-model of the video feature extraction model to obtain the image features of the target video. The embedding layer of the feature fusion sub-model processes the text information to obtain the text features of the target video, and the image features and the text features are feature-fused through the feature fusion layer of the feature fusion sub-model to obtain the fusion of the target video. feature;

The fusion feature is processed by the text generation sub-model of the video feature extraction model, a plurality of characters satisfying the text generation conditions are output, and the content description text of the target video is generated based on the plurality of characters.

In the embodiment of the present disclosure, by constructing an image feature extraction sub-model in the video feature extraction model, the image features of the target video can be accurately extracted, and by constructing a feature fusion sub-model in the video feature extraction model, not only the target video can be obtained The text features of the video can also feature fusion of the image features and text features of the target video, so that the subsequent processing based on the fusion features can output multiple characters that meet the text generation conditions, and then automatically generate based on the output multiple characters. The content description text of the target video provides a video feature extraction model based on text generation. The generated content description text contains rich information, which can better characterize the target video and improve the accuracy of video representation. .

In some embodiments, the method further includes:

Image reconstruction is performed on the image features in the fusion feature through the image reconstruction sub-model of the video feature extraction model to obtain the image reconstruction feature of the original image size of the target video.

According to a third aspect of the embodiments of the present disclosure, there is provided a training device for a video feature extraction model, the device comprising:

an acquisition unit, configured to perform acquisition of image information, text information, image tags and text tags of the sample video, where the image tag represents an image reconstruction feature, and the text tag represents the content description text of the sample video;

The input unit is configured to input the image information and the text information into the video feature extraction model, perform feature extraction on the image information through the image feature extraction sub-model of the video feature extraction model, and obtain the image feature of the sample video, through The text information is processed by the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain the text features of the sample video, and the image features and the text features are feature-fused through the feature fusion layer of the feature fusion sub-model, Obtain the fusion feature of the sample video;

The processing unit is configured to perform image restoration through the image reconstruction sub-model of the video feature extraction model to the image features in the fusion feature to obtain an image training result of the original image size, and generate the sub-model through the text of the video feature extraction model. The fusion feature is processed to obtain the text training result;

A training unit configured to perform image labeling and text labeling based on the image training result, the text training result and the sample video, and adjust the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text Generate the model parameters of the sub-model to train the video feature extraction model.

In some embodiments, the obtaining unit is configured to perform at least one of the following:

Obtain the cover image of the sample video; or, obtain at least one frame of image in the sample video.

In some embodiments, the obtaining unit is configured to perform at least one of the following:

Obtain the description information of the sample video; obtain the title information of the sample video; obtain the subtitle information of the sample video; Identify the obtained result; obtain the audio recognition result of the sample video, where the audio recognition result is the result obtained by performing audio recognition on the background audio of the sample video.

In some embodiments, the content description text is at least one of content category description text, content form description text, content topic description text, and content detail description text.

In some embodiments, the input unit includes a processing subunit configured to perform any of the following:

Through the self-attention layer included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video;

Through the deep belief network included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video.

In some embodiments, the processing unit includes a text generation sub-unit configured to perform:

Through the self-attention layer included in the text generation sub-model, the fusion feature is processed to obtain the text training result.

In some embodiments, the text training result includes multiple types of content description text;

The device also includes an adding unit configured to perform on the fusion feature, adding type identifiers of various types;

The processing unit includes a text generation sub-unit, and is further configured to input the fusion feature added with the type identifier into the text generation sub-model, and through the text generation sub-model, based on the processing mechanisms corresponding to each type identifier, the fusion The features are processed to obtain the multiple types of content description texts.

In some embodiments, the number of the text information is multiple;

The device also includes a splicing unit configured to perform splicing of a plurality of the text information to obtain the spliced text information;

The input unit is further configured to perform the step of processing the text information through the embedding layer of the feature fusion sub-model of the video feature extraction model based on the spliced text information to obtain text features of the sample video.

In some embodiments, the input unit is further configured to perform:

From the spliced text information, extract the characters of the previous target number;

Based on the extracted characters, the step of processing the text information through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain text features of the sample video is performed.

In some embodiments, the training unit includes:

The determining subunit is configured to perform in the ith iteration process of model training, based on the image training result of the ith iteration process, the text training result and the image label and text label of the sample video, determine the ith time The model loss value of the iterative process, the i is a positive integer greater than 1;

The adjustment subunit is configured to execute the model loss value based on the i-th iteration process, adjust the model parameters of the video feature extraction model determined in the i-1-th iteration process, and repeat the above-mentioned iterative process of training until training meet the target conditions.

In some embodiments, the determining subunit includes:

The image reconstruction loss value determination subunit is configured to execute the image training result based on the i-th iteration process and the image label of the sample video, and determine the image reconstruction loss value of the i-th iteration process, and the image reconstruction loss value represents the difference between the training result of the image and the label of the image;

The text generation loss value determination subunit is configured to execute the text generation loss value of the ith iteration process based on the text training result of the ith iteration process and the text label of the sample video, and the text generation loss value represents the difference between the text training result and the text label;

The model loss value determination subunit is configured to perform the reconstruction loss value based on the image and the text generation loss value, and determine the model loss value of the ith iteration process.

In some embodiments, the model loss value determination subunit is configured to perform:

Based on the image reconstruction loss value, the weight coefficient corresponding to the image reconstruction loss value, the text generation loss value, and the weight coefficient corresponding to the text generation loss value, a weighted sum is performed to obtain the model loss value of the i-th iteration process.

In some embodiments, the text training result includes multiple types of content description text;

The text generation loss value determination subunit is configured to perform:

For any type, determine the loss value of the i-th iteration process on the type based on the text training result of the i-th iteration process on the type and the text label of the sample video on the type;

Based on the loss values of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, weighted summation is performed to obtain the text generation loss value of the i-th iteration process.

In some embodiments, the apparatus further includes a determining unit configured to perform:

For any type, based on the correct number of texts on the type in the ith iteration and the total number of texts, determine the correct scale of the ith iteration on the type, the correct scale representing the ith iteration The ratio of the number of correct texts to the total number of texts in the process;

Based on the correct scale of the ith iterative process on the type, the weight coefficient of the video feature extraction network on the type is determined, wherein the correct scale is negatively correlated with the weight coefficient.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a text generation device based on a video feature extraction model, where the video feature extraction model is trained based on the first aspect or the training method shown in any embodiment of the first aspect. , the device includes:

an acquisition unit, configured to perform acquisition of image information and text information of the target video;

The input unit is configured to input the image information and the text information into the video feature extraction model, and perform feature extraction on the image information through the image feature extraction sub-model of the video feature extraction model to obtain the image feature of the target video, The text information is processed through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain the text features of the target video, and the image features and the text features are feature-fused through the feature fusion layer of the feature fusion sub-model. , to obtain the fusion feature of the target video;

The processing unit is configured to process the fusion feature through the text generation sub-model of the video feature extraction model, output a plurality of characters satisfying the text generation condition, and generate a content description text of the target video based on the plurality of characters.

In some embodiments, the processing unit is further configured to execute:

Image reconstruction is performed on the image features in the fusion feature through the image reconstruction sub-model of the video feature extraction model to obtain the image reconstruction feature of the original image size of the target video.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer device, the computer device comprising:

one or more processors;

memory for storing executable program code for the processor;

Wherein, the processor is configured to execute the program code to implement the above-mentioned first aspect or the method for training a video feature extraction model shown in any embodiment of the first aspect, or the above-mentioned second aspect or the second aspect The text generation method based on the video feature extraction model shown in any one of the embodiments.

According to a sixth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, the computer-readable storage medium comprising: when program code in the computer-readable storage medium is executed by a processor of a computer device, causing the computer device A training method capable of performing the video feature extraction model shown in the first aspect or any embodiment of the first aspect, or the video-based feature extraction model shown in any embodiment of the second aspect or the second aspect text generation method.

According to a seventh aspect of the embodiments of the present disclosure, there is provided a computer program product, including a computer program, which, when executed by a processor, implements the video feature extraction according to the first aspect or any one of the embodiments of the first aspect. A training method for a model, or a text generation method based on a video feature extraction model shown in the second aspect or any embodiment of the second aspect.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure and do not unduly limit the present disclosure.

1 is a schematic diagram of an implementation environment of a method for training a video feature extraction model according to an exemplary embodiment;

2 is a flowchart of a method for training a video feature extraction model according to an exemplary embodiment;

3 is a flow chart of a text generation method based on a video feature extraction model according to an exemplary embodiment;

4 is a flowchart of a method for training a video feature extraction model according to an exemplary embodiment;

5 is a framework of a video feature extraction model according to an exemplary embodiment;

6 is a flowchart of a text generation method based on a video feature extraction model according to an exemplary embodiment;

FIG. 7 is a block diagram of a training apparatus for a video feature extraction model according to an exemplary embodiment;

FIG. 8 is a block diagram of a text generation apparatus based on a video feature extraction model according to an exemplary embodiment;

FIG. 9 is a block diagram of a terminal according to an exemplary embodiment;

Fig. 10 is a block diagram of a server according to an exemplary embodiment.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure, and Signals are authorized by users or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the image information and text information involved in the embodiments of the present disclosure are obtained under the condition of full authorization.

In some embodiments, the terminal is provided with a permission query page, which is used to ask the user whether to grant the user permission to obtain the image information and text information of the video. On the permission query page, an authorization authorization control and an authorization rejection control are displayed, In the case where the user's triggering operation on the consent authorization control is detected, the training method of the video feature extraction model provided by the embodiment of the present disclosure is used to obtain the image information and text information of the sample video, and then based on the image information of the sample video and text information to train the video feature extraction model.

FIG. 1 is a schematic diagram of an implementation environment of a training method for a video feature extraction model according to an exemplary embodiment. Referring to FIG. 1 , the implementation environment includes: a terminal 101 and a server 102 .

The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop portable computer. The terminal 101 has a communication function and can access a wired network or a wireless network. The terminal 101 may generally refer to one of multiple terminals, and this embodiment only takes the terminal 101 as an example for illustration. Those skilled in the art may know that the number of the above-mentioned terminals may be more or less.

The server 102 may be an independent physical server, or a server cluster or a distributed file system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, and cloud communications , middleware services, domain name services, security services, CDN (Content Delivery Network), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. In some embodiments, the server 102 and the terminal 101 are directly or indirectly connected through wired or wireless communication, which is not limited in this embodiment of the present disclosure. Optionally, the number of the foregoing servers 102 may be more or less, which is not limited in this embodiment of the present disclosure. Of course, the server 102 may also include other functional servers in order to provide more comprehensive and diversified services.

In some embodiments, the training method of the video feature extraction model provided by the embodiment of the present disclosure is performed by the terminal 101. For example, the terminal 101 uses the video provided by the embodiment of the present disclosure in response to the training operation of the video feature extraction model. The training method of the feature extraction model, the model training is performed on the video feature extraction model; in some other embodiments, the training method of the video feature extraction model provided by the embodiments of the present disclosure is performed jointly by the terminal 101 and the server 102, for example, the terminal 101 sends the training data of the video feature extraction model to the server 102 in response to the uploading operation of the training data of the video feature extraction model, then the server 102 receives the training data uploaded by the terminal 101, and uses the video provided by the embodiment of the present disclosure. The training method of the feature extraction model is to perform model training on the video feature extraction model.

It should be noted that, the video feature extraction model provided by the embodiments of the present disclosure can be applied in the scenarios of video recommendation, video classification, or video search. In some embodiments, the video feature extraction model trained by the embodiments of the present disclosure is used to obtain the content description text of the video, and then the functions of video recommendation, video classification or video search are implemented by using the obtained content description text.

Fig. 2 is a flow chart of a method for training a video feature extraction model according to an exemplary embodiment. As shown in Fig. 2, the method is executed by a computer device, and the computer device can be provided as shown in Fig. 1 above. Terminal or server, schematically, the method includes the following steps:

In step 201, the computer device acquires image information, text information, image label and text label of the sample video, where the image label represents the image reconstruction feature, and the text label represents the content description text of the sample video.

In step 202, the computer equipment inputs the image information and the text information into the video feature extraction model, and performs feature extraction on the image information through the image feature extraction sub-model of the video feature extraction model to obtain the image features of the sample video, The text information is processed by the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain the text features of the sample video, and the image features and the text features are feature-fused through the feature fusion layer of the feature fusion sub-model, Get the fusion features of the sample video.

In step 203, the computer equipment performs image restoration on the image features in the fusion feature through the image reconstruction sub-model of the video feature extraction model to obtain an image training result of the original image size, and generates a sub-model through the text of the video feature extraction model. The fusion feature is processed to obtain the text training result.

In step 204, the computer device adjusts the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text based on the image training result, the text training result and the image label and text label of the sample video Generate the model parameters of the sub-model to train the video feature extraction model.

In the technical solution provided by the embodiments of the present disclosure, the image information and text information of the sample video and the text label and image label of the sample video are used to perform model training on the video feature extraction model. The feature extraction sub-model can accurately extract the image features of the sample video. By constructing the feature fusion sub-model in the video feature extraction model, not only the text features of the sample video can be obtained, but also the image features and text of the sample video. Feature fusion is performed on features, so that based on the fusion features, on the one hand, image reconstruction can be performed on the sample video to obtain high-quality image features, and on the other hand, the content description text of the sample video can also be generated. A model training method based on dual training tasks. In the case where the text generation task is the main task and the image reconstruction task is the auxiliary task, since the image label of the sample video represents the image reconstruction feature, in the process of model training, it can be Improve the ability of the video feature extraction model to extract image features, and then obtain high-quality image features. On the basis of obtaining high-quality image features, a video feature extraction model with better text generation ability can be trained to improve The training effect of the video feature extraction model.

In some embodiments, the acquisition process of the image information of the sample video includes at least one of the following:

Obtain the cover image of the sample video; or, obtain at least one frame of image in the sample video.

In some embodiments, the process of acquiring the text information of the sample video includes at least one of the following:

Obtain the description information of the sample video; obtain the title information of the sample video; obtain the subtitle information of the sample video; Identify the obtained result; obtain the audio recognition result of the sample video, where the audio recognition result is the result obtained by performing audio recognition on the background audio of the sample video.

In some embodiments, the content description text is at least one of content category description text, content form description text, content topic description text, and content detail description text.

In some embodiments, feature fusion is performed on the image feature and the text feature through the feature fusion layer of the feature fusion sub-model to obtain the fusion feature of the sample video including any of the following:

Through the self-attention layer included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video;

Through the deep belief network included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video.

In some embodiments, the fusion feature is processed by the text generation sub-model of the video feature extraction model, and the obtained text training result includes:

Through the self-attention layer included in the text generation sub-model, the fusion feature is processed to obtain the text training result.

In some embodiments, the text training result includes multiple types of content description text;

The fusion feature is processed by the text generation sub-model of the video feature extraction model to obtain the text training result, the method further includes:

On the fusion feature, add the type identifier of each type;

The fusion feature is processed by the text generation sub-model of the video feature extraction model, and the text training result obtained includes:

The fusion feature after adding the type identifier is input into the text generation sub-model, and the fusion feature is processed based on the processing mechanism corresponding to each type identifier through the text generation sub-model to obtain the multiple types of content description texts.

In some embodiments, the number of the text information is multiple;

The text information is processed by the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain the text feature of the sample video, the method further includes:

Splicing a plurality of the text information to obtain the spliced text information;

Based on the spliced text information, the step of processing the text information through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain text features of the sample video is performed.

In some embodiments, after splicing a plurality of the text information to obtain the spliced text information, the method further includes:

From the spliced text information, extract the characters of the previous target number;

Based on the extracted characters, the step of processing the text information through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain text features of the sample video is performed.

In some embodiments, the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text generation sub-model are adjusted based on the image training result, the text training result and the image labels and text labels of the sample video The model parameters of the sub-model to train this video feature extraction model include:

In the ith iteration process of model training, the model loss value of the ith iteration process is determined based on the image training results and text training results of the ith iteration process and the image labels and text labels of the sample video. i is a positive integer greater than 1;

Based on the model loss value of the i-th iterative process, adjust the model parameters of the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text generation sub-model determined in the i-1-th iteration process , and repeat the iterative process of the above training until the training meets the target condition.

In some embodiments, determining the model loss value of the i-th iteration process based on the image training results and text training results of the i-th iteration process and the image labels and text labels of the sample video includes:

Based on the image training result of the ith iteration process and the image label of the sample video, the image reconstruction loss value of the ith iteration process is determined, and the image reconstruction loss value represents the difference between the image training result and the image label ;

Based on the text training result of the ith iteration process and the text label of the sample video, determine the text generation loss value of the ith iteration process, where the text generation loss value represents the difference between the text training result and the text label ;

Based on the image reconstruction loss value and the text generation loss value, the model loss value of the i-th iteration process is determined.

In some embodiments, based on the image reconstruction loss value and the text generation loss value, determining the model loss value of the i-th iteration process includes:

Based on the image reconstruction loss value, the weight coefficient corresponding to the image reconstruction loss value, the text generation loss value, and the weight coefficient corresponding to the text generation loss value, a weighted sum is performed to obtain the model loss value of the i-th iteration process.

In some embodiments, the text training result includes multiple types of content description text;

Based on the text training result of the ith iterative process and the text label of the sample video, determining the text generation loss value of the ith iterative process includes:

For any type, determine the loss value of the i-th iteration process on the type based on the text training result of the i-th iteration process on the type and the text label of the sample video on the type;

Based on the loss values of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, weighted summation is performed to obtain the text generation loss value of the i-th iteration process.

In some embodiments, based on the loss value of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, a weighted sum is performed to obtain the i-th iteration process Before generating the loss value for the text of , the method also includes:

For any type, based on the correct number of texts on the type and the total number of texts in the ith iteration process, determine the correct scale of the ith iteration process on the type, the correct scale representing the ith iteration process The ratio of the number of correct texts to the total number of texts in the process;

Based on the correct scale of the i-th iterative process on the type, the weight coefficient of the video feature extraction network on the type is determined, wherein the correct scale is negatively correlated with the weight coefficient.

Fig. 3 is a flow chart of a text generation method based on a video feature extraction model according to an exemplary embodiment. As shown in Fig. 3, the method is executed by a computer device, and the computer device can be provided as shown in Fig. 1 above. The terminal or server that is sent out, schematically, the method includes the following steps:

In step 301, the computer device acquires image information and text information of the target video.

In step 302, the computer equipment inputs the image information and the text information into the video feature extraction model, and performs feature extraction on the image information through the image feature extraction sub-model of the video feature extraction model to obtain the image feature of the target video, The text information is processed through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain the text features of the target video, and the image features and the text features are feature-fused through the feature fusion layer of the feature fusion sub-model. , to obtain the fusion features of the target video.

In step 303, the computer device processes the fusion feature through the text generation sub-model of the video feature extraction model, outputs a plurality of characters satisfying the text generation condition, and generates a content description text of the target video based on the plurality of characters.

According to the technical solutions provided by the embodiments of the present disclosure, by constructing an image feature extraction sub-model in the video feature extraction model, the image features of the target video can be accurately extracted, and by constructing a feature fusion sub-model in the video feature extraction model, not only can To the text features of the target video, it can also perform feature fusion on the image features and text features of the target video, so that the subsequent processing based on the fusion features can output multiple characters that meet the text generation conditions, and then based on the output multiple characters The content description text of the target video can be automatically generated, and a video feature extraction model based on text generation is provided. The generated content description text contains a wealth of information, which can better characterize the target video and improve the video representation. accuracy.

In some embodiments, the method further includes:

Image reconstruction is performed on the image features in the fusion feature through the image reconstruction sub-model of the video feature extraction model to obtain the image reconstruction feature of the original image size of the target video.

The above-mentioned FIG. 2 to FIG. 3 are only the basic flow of the present disclosure. The solution provided by the present disclosure will be further elaborated below based on a specific implementation manner. FIG. 4 is a video feature extraction according to an exemplary embodiment. For the flowchart of the training method of the model, see Figure 4, the method includes:

In step 401, the computer device acquires image information, text information, image label and text label of the sample video, where the image label represents the image reconstruction feature, and the text label represents the content description text of the sample video.

In the embodiments of the present disclosure, sample videos are used to refer to training videos used to train the video feature extraction model. In some embodiments, the number of sample videos is multiple. It should be understood that the image information, text information, image labels and text labels of the sample video are obtained as a training data set.

In some embodiments, the process of acquiring the image information of the sample video by the computer device includes at least one of the following: acquiring a cover image of the sample video; or, acquiring at least one frame of image in the sample video.

Wherein, the number of the at least one frame of images may be one frame, two frames or more than two frames, such as three frames. In some embodiments, the process of acquiring the at least one frame of image by the computer device is: randomly select a preset number of images from the multiple frames of images included in the sample video, for example, randomly select three frames of images; or, from the sample video From the multiple frames of images included in the video, a preset number of images are uniformly extracted, for example, three frames of images are extracted at equal intervals; key frames are extracted from the multiple frames of images included in the sample video. Certainly, the computer device can also acquire at least one frame of image in the sample video based on other manners, which is not limited in this embodiment of the present disclosure.

In the above embodiment, by obtaining the cover image of the sample video or the image frame included in the sample video, the image information of the sample video can be quickly obtained, which not only ensures the efficiency of obtaining image information, but also enriches the types of image information. , which improves the flexibility of acquiring image information.

In some embodiments, the process of acquiring the text information of the sample video by the computer device includes at least one of the following: acquiring description information of the sample video; acquiring title information of the sample video; acquiring subtitle information of the sample video; acquiring the sample The text recognition result of the video; obtain the audio recognition result of the sample video.

Wherein, the description information is information used to describe the video topic of the sample video, such as theme description information or topic description information (caption); or, the description information is information used to describe the video content of the sample video, such as content description information (hashtag). The title information refers to the video title (title) of the sample video. In some embodiments, the description information and title information are information set by the publisher of the sample video. For example, when the publisher of the sample video publishes the sample video, the terminal corresponding to the publisher is provided with a description information input box and Title information entry box, through the description information entry box and the title information entry box, the publisher can set the description information and title information of the sample video, and then, the computer equipment can also obtain the sample video while obtaining the sample video. Description and title information for this sample video.

The subtitle information refers to the subtitle (text) included in the image in the sample video. In some embodiments, the computer device uses a subtitle extraction tool to extract the subtitle information of the sample video. The text recognition result is a result obtained by performing text recognition on at least one frame of image in the sample video. In some embodiments, the computer device uses OCR (Optical Character Recognition, Optical Character Recognition) technology to perform text recognition on the sample video. The frame image is used for text recognition, and the text recognition result of the sample video is obtained. The audio recognition result is the result obtained by performing audio recognition on the background audio of the sample video. In some embodiments, the computer equipment utilizes ASR (Automatic Speech Recognition, automatic speech recognition) technology to perform audio recognition on the background audio of the sample video, Get the audio recognition result of the sample video.

In the above-mentioned embodiment, by obtaining the description information, title information, subtitle information, text recognition result or audio recognition result of the sample video, the text information of the sample video can be quickly obtained. The types of text information are enriched, and the flexibility of obtaining text information is improved.

The image tag represents the image reconstruction feature of the sample video after image reconstruction, wherein the image reconstruction refers to performing image restoration or image restoration on the image in the video to obtain complete image features. The text tag represents the content description text of the sample video, and the content description text refers to a sentence used to describe the content of the sample video.

In some embodiments, the content description text is at least one of content category description text, content form description text, content topic description text, and content detail description text.

In some embodiments, the content category description text includes multi-level category description text, for example, a first-level category description text and a second-level category description text, wherein the first-level category description text refers to a description text for describing a video. The sentence of the first-level category, and the description text of the second-level category refers to the sentence used to describe the second-level category of the video. It should be understood that the first-level category refers to the general classification of the video, and the second-level category refers to the sub-category of the video on the basis of the first-level category, wherein the second-level category is a tree structure relative to the first-level category, That is to say, a first-level category will include multiple second-level categories. For example, the first-level category may be life category, the second-level category may be life records, good things sharing or health preservation, etc., or, the first-level category may be beauty makeup, and the second-level category may be beauty makeup teaching. , beauty reviews, skin care and more.

The content form description text refers to a sentence used to describe the content form of the video, and the content form represents the presentation form (or called the shooting form) of the video. Illustratively, the content format may be a documentary format, a sitcom format, a street interview format, or the like. The content topic description text refers to a sentence used to describe the content topic of the video, where the content topic represents the content topic of the video, for example, the video topic may be a video topic. The content details description text refers to a sentence for describing the content details of the video, for example, a sentence for describing the screen content of the video.

In the above embodiment, by setting multiple types of content description texts, on the one hand, more expressive content description texts can be generated; The content is described, which enriches the type of the generated content description text, and can more fully and completely characterize the video.

In step 402, the computer device inputs the image information and the text information into the video feature extraction model.

In the embodiment of the present disclosure, the video feature extraction model provides the functions of image reconstruction and text generation. In some embodiments, the video feature extraction model is an Encoder-Decoder architecture based on a Self Attention Mechanism. Among them, the self-attention mechanism is a mechanism that learns the meaning of features based on the dependencies between features. In the self-attention mechanism, for each input feature, the similarity between the feature and its neighboring features is calculated according to the feature. Or correlation, such as calculating the vector dot product of the two, calculating the vector similarity of the two, or by introducing an additional neural network to evaluate, etc., to obtain the calculated score of the feature and its adjacent features, and then use functions such as softmax ( The calculation method of an activation function) performs numerical conversion on the calculated score. In this way, on the one hand, the calculated score can be converted into a probability distribution with the sum of the element weights being 1, which realizes normalization. On the other hand, through The internal mechanism of the softmax function can highlight the weight of important elements, and then use the weight of each element to perform a weighted summation to output the self-attention score.

Exemplarily, FIG. 5 is a block diagram of a video feature extraction model according to an exemplary embodiment. Referring to FIG. 5 , the video feature extraction model includes an image feature extraction sub-model, a feature fusion sub-model, an image reconstruction sub-model, and a text generation sub-model. The sub-model, wherein the encoder of the video feature extraction model includes a feature fusion sub-model, and the decoder of the video feature extraction model includes a text generation sub-model. The following describes the training method of the video feature extraction model provided by the embodiment of the present disclosure based on the video feature extraction model shown in FIG. 5 .

In step 403, feature extraction is performed on the image information through an image feature extraction sub-model of the video feature extraction model to obtain image features of the sample video.

In the embodiment of the present disclosure, the image feature extraction sub-model provides a function of extracting image features of a video. In some embodiments, the image feature extraction sub-model is Resnet (residual network) or ViT (Vision Transformer, visual transformer) or Swin Tiny model.

In some embodiments, after the computer device inputs the image information and the text information into the video feature extraction model, the image information is input into the image feature extraction sub-model of the video feature extraction model through the video feature extraction model, and the image feature extraction model is used to extract the image information. The extraction sub-model performs feature extraction on the image information, and can obtain image features of predetermined dimensions, such as image features of 512 dimensions (or other quantitative dimensions).

In step 404, the text information is processed through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain text features of the sample video.

Among them, the embedding layer is used to convert the numerical value into a vector with a fixed size. In some embodiments, after inputting the image information and the text information into a video feature extraction model, the computer device inputs the text information into a feature fusion sub-model of the video feature extraction model through the video feature extraction model, and uses the feature fusion sub-model The embedding layer of the model processes the text information, and can obtain text features of predetermined dimensions, such as text features of 512 dimensions (or other quantity dimensions). It should be noted that the dimension of the text feature is the same as the dimension of the image feature.

In some embodiments, the number of text information is multiple (such as two or more). Accordingly, before the feature extraction is performed on the text information, the method further includes: splicing the multiple text information to obtain For the spliced text information, based on the spliced text information, the above step 404 is performed. Illustratively, taking the five text information shown in step 401 as an example, the description information, title information, subtitle information, text recognition result and audio recognition result of the sample video are spliced to obtain the spliced text information, and then The spliced text information is input into the above-mentioned text feature extraction sub-model to execute the above-mentioned step 404 . In this embodiment, when the number of text information is multiple, the spliced text information is obtained by splicing the text information, and then the process of extracting text features is performed by using the spliced text information. Various types of text information improve the accuracy of extracting text features.

Further, in some embodiments, after obtaining the spliced text information, the method further includes: extracting the characters of the previous target number from the spliced text information, and performing the above step 404 based on the extracted characters. The target number is a preset fixed number, such as 200. In this embodiment, in the spliced text information, the characters of the previous target number are extracted, so that the subsequent process of extracting text features can be performed based on the extracted certain number of characters, and on the basis of ensuring that sufficient text information is input , which reduces the computational complexity of the video feature extraction model and improves the efficiency of text feature extraction.

In step 405, feature fusion is performed on the image feature and the text feature through the feature fusion layer of the feature fusion sub-model to obtain the fusion feature of the sample video.

Among them, the feature fusion sub-model provides the function of feature fusion of image features and text features to output features with more video representation capabilities. In some embodiments, the feature fusion layer is provided as a self-attention layer, such as a Transformer layer based on a self-attention mechanism. Accordingly, through the self-attention layer included in the feature fusion sub-model, the The image feature and the text feature are processed to obtain the fusion feature of the sample video; in other embodiments, the feature fusion layer is provided as a deep belief network, and correspondingly, through the deep belief network included in the feature fusion sub-model, The image feature and the text feature are processed to obtain the fusion feature of the sample video. In this embodiment, by setting the self-attention layer deep belief network in the feature fusion sub-model, and then using the self-attention mechanism deep belief network to perform feature fusion, it is possible to obtain features with more video representation capabilities and improve feature fusion. accuracy. Of course, in other embodiments, other network layers with a feature fusion function can also be set in the feature fusion sub-model to implement the feature fusion function, which is not limited in the embodiments of the present disclosure.

In the embodiment of the present disclosure, the feature fusion sub-model includes an embedding layer (Embedding layer) and a feature fusion layer (such as a multi-layer self-attention layer or a deep belief network), so as to use the embedding layer of the feature fusion sub-model to extract sample videos and then combine the image features output by the image feature extraction sub-model, and then use the feature fusion layer of the feature fusion sub-model to perform feature fusion. The flow architecture can more fully fuse the features of different modalities and improve the accuracy of feature fusion.

In step 406, image restoration is performed on the image features in the fusion feature through the image reconstruction sub-model of the video feature extraction model to obtain an image training result of the original image size.

The image reconstruction sub-model is provided with the function of performing image reconstruction processing on the video to output the image reconstruction features of the video. In some embodiments, the image reconstruction sub-model is provided with the function of performing image restoration on the images in the video, so as to Output image reconstruction features at the original image size. Image restoration is the process of reconstructing a degraded image (or called a degraded image) to restore the degraded image to the original image. The original image size is the size of the image in the video, and in some embodiments, the original image size is determined based on horizontal pixels, vertical pixels in the image, and color information in the image. In this way, by restoring the image features of the sample video to the image reconstruction features of the original image size, high-quality image reconstruction features are obtained, and then the high-quality image reconstruction features are subsequently used for the process of model training. The image training result is the image reconstruction feature obtained during the model training process. In some embodiments, the image reconstruction sub-model includes multiple MLP networks (multi-layer neural networks) or other network layers with image reconstruction functions, which are not limited in this embodiment of the present disclosure.

In some embodiments, a preset number of image features are extracted from the fusion features output by the feature fusion sub-model, and the extracted image features are input into the image reconstruction sub-model of the video feature extraction model, and then the video features The image reconstruction sub-model of the model is extracted, the extracted image features are processed, and the image reconstruction features of the sample video are output, that is, the image training result is obtained. Among them, the preset number refers to the number of image features. It should be noted that since the feature fusion layer of the feature fusion sub-model includes multiple layers of self-attention layers, and the self-attention layer can output a certain number of features under the premise of inputting a certain number of features. The same number of features, in this way, extracting image features according to the preset number can extract sufficient image features, and since the image features and text features have undergone the feature fusion sub-model, the output image features are already fused with text. The characteristics of the feature, that is, the accuracy of image reconstruction is improved.

In step 407, the fusion feature is processed by the text generation sub-model of the video feature extraction model to obtain a text training result.

Among them, the text generation sub-model provides the function of performing text generation processing on the video, so as to output the content description text of the video. The text training result is the content description text generated during the model training process.

In some embodiments, the text generation sub-model includes multiple layers of self-attention layers. Accordingly, through the video feature extraction model, the fusion feature is input into the text generation sub-model of the video feature extraction model, and the text generation sub-model through the text generation sub-model The included self-attention layer processes the fusion feature to obtain the text training result. In some embodiments, the fusion feature is processed through the multi-layer self-attention layer included in the text generation sub-model, and a plurality of characters whose self-attention scores meet the text generation conditions are output, based on the output of the plurality of characters Characters generate the content description text of the sample video, that is, the text training result is obtained. For example, the text generation condition may be that the self-attention score reaches a score threshold.

Based on the multiple types of content description texts shown in step 401, in an optional embodiment, the text generation sub-model is provided with the function of performing multiple types of text generation tasks. The multi-layer self-attention layers included in the text generation sub-model respectively process the fusion features according to the processing mechanisms of the text generation tasks corresponding to the multiple types to generate multiple types of content description texts of the sample video. Further, for any type, through the multi-layer self-attention layer included in the text generation sub-model, the first segment of the feature sequence in the fusion feature is processed, and the self-attention score in the first segment of the feature sequence is output. Characters for text generation conditions; based on the features output in the first feature sequence, continue to process the second feature sequence in the fusion feature, and output the characters whose self-attention score in the second feature sequence reaches the text generation conditions; Based on the features output in the first feature sequence and the features output in the second feature sequence, continue to process the third feature sequence in the fusion feature, and output the self-attention score in the third feature sequence to achieve text generation Conditional characters, and then, based on the output characters, repeat the above processing and output processes, output the characters whose self-attention score reaches the text generation conditions in the next feature sequence, and obtain a plurality of text generation sub-model output. character, splicing according to the output sequence of the multiple characters to obtain the content description text of the sample video, that is, the above text training result is obtained.

In the above embodiment, by setting a self-attention layer in the text generation sub-model, and then using the self-attention mechanism to generate content description text, the accuracy of text generation is improved.

In some embodiments, the text generation sub-model is further provided with a masking mechanism, which means to block the features that do not need attention in a feature sequence, so as to avoid the influence on the video feature extraction model and improve the self-attention mechanism. accuracy.

In some embodiments, when the text generation sub-model is used to perform multiple types of text generation tasks, before inputting the fused feature into the text generation sub-model, the method further includes: on the fused feature, adding various The type identification of the type is performed based on the fusion feature after adding the type identification, and the above step 407 is performed. The type identifier is used to indicate the text generation task of the corresponding type. In some embodiments, through the video feature extraction model, the fusion feature after adding the type identifier is input into the text generation sub-model, and the fusion feature is processed based on the processing mechanism corresponding to each type identifier through the text generation sub-model. to obtain the content description texts of the multiple types. In this embodiment, by adding each type of type identifier to the fusion feature, the text generation sub-model in the video feature extraction model can trigger the generation of the content description text of each type of sample video based on each type of type identifier. , to ensure smooth text generation.

It should be noted that the above steps 406 to 407 take the example of outputting the image training results first and then outputting the text training results. In other embodiments, the computer device can also output the text training results first and then output the image training results, or the computer The device can also output the image training result and the text training result at the same time, and the embodiment of the present disclosure does not limit the execution order of step 406 and step 407 .

In step 408, the computer device adjusts the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text based on the image training result, the text training result and the image label and text label of the sample video Generate the model parameters of the sub-model to train the video feature extraction model.

For the above steps 402 to 408, in some embodiments, the computer device inputs the image information and text information of the sample video into the initial video feature extraction model during the first iteration of model training, and triggers the video feature extraction model. The model executes the processing process of the above-mentioned steps 402 to 407, and obtains the image training result and the text training result of the first iterative process, based on the image training result of the first iterative process, the text training result and the image label of the sample video, Text label, adjust the model parameters of the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text generation sub-model in the initial video feature extraction model; in the adjusted video feature extraction model If the target condition is not met, the next iteration process is performed based on the adjusted model parameters, and then, in the i-th iteration process of model training, the image information and text information of the sample video are input for the i-1th time. The video feature extraction model determined by the iterative process triggers the video feature extraction model to execute the processing process of the above-mentioned steps 402 to 407, and then obtains the image training result and the text training result of the i-th iteration process, based on the i-th iteration process The image training result, text training result and the image label and text label of the sample video, adjust the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and The model parameters of the text generation sub-model, in the case that the adjusted video feature extraction model does not meet the target conditions, the i+1 th iteration process is performed based on the adjusted model parameters, and the above-mentioned iterative process of training is repeated until the training To satisfy the target condition, the i is a positive integer greater than 1.

In some embodiments, the target condition satisfied by the training is that the number of training iterations of the video feature extraction model reaches the target number of times, and the target number of times is a preset number of training iterations, such as 1000 times; or, the target condition satisfied by the training is the model loss The value satisfies the target threshold condition, such as the loss value is less than 0.00001. The embodiment of the present disclosure does not limit the setting of the target condition.

For the above process of adjusting the model parameters of the video feature extraction model, in some embodiments, in the ith iteration process of model training, based on the image training results, text training results and the sample of the ith iteration process The image label and text label of the video are used to determine the model loss value of the i-th iteration process; based on the model loss value of the i-th iteration process, the model parameters of the video feature determined in the i-1th iteration process are extracted. make adjustments. Based on steps 408A to 408C, the process of determining the model loss value of the i-th iteration process by the computer device will be described below:

In step 408A, the computer device determines the image reconstruction loss value of the i-th iteration process based on the image training result of the i-th iteration process and the image label of the sample video, and the image reconstruction loss value represents the image training result and the image reconstruction loss value. The difference between the image tags.

In some embodiments, the computer device determines the MSELoss (Mean Square Eerror Loss, mean square error loss) value of the ith iterative process based on the image training result of the ith iterative process and the image label of the sample video, and sets the The determined MSELoss value is used as the image reconstruction loss value.

In step 408B, the computer device determines the text generation loss value of the ith iteration process based on the text training result of the ith iteration process and the text label of the sample video, and the text generation loss value indicates that the text training result is different from The difference between the text labels.

In some embodiments, when the text training result includes multiple types of content description texts, for any type, the text training result on the type based on the i-th iteration process and the sample video in the type The text label on , determine the loss value of the i-th iteration process on the type; based on the loss value of the i-th iteration process on the multiple types and the weight of the video feature extraction network on the multiple types coefficients, and perform weighted summation to obtain the text generation loss value of the i-th iteration process.

In some embodiments, for any type, the computer device determines, based on the text training result of the i-th iteration process on the type and the text label of the sample video on the type On the CEloss (Cross Entropy Loss, cross entropy loss) value, based on the CEloss value of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, perform weighted summation , the text generation loss value of the i-th iteration process is obtained.

Regarding the above CEloss value, in some embodiments, the process of determining the CEloss value by the computer device includes: for any type, the computer device is based on the text training result of the i-th iteration process on the type, the sample video on the type The text labels of , the number of sample videos, and the CEloss formula (1), determine the cross-entropy loss value of the i-th iteration process on this type.

In the formula, CEloss represents the cross entropy loss value; m represents the number of sample videos in the training data set; y _k represents the text training result of the video feature extraction model for the kth sample video; p represents the video feature extraction model for the kth sample video. The correct probability of the text training result of the sample video, for example, the correct probability can be the similarity between the text training result and the text label.

Regarding the weight coefficients corresponding to the above multiple types, in some embodiments, the process of determining the weight coefficient by the computer device includes: for any type, based on the correct number of texts on the type and the total number of texts in the i-th iteration process, Determine the correct proportion of the ith iteration process on this type, the correct proportion represents the proportion of the correct text quantity to the total text quantity in the ith iteration process, based on the correctness of the ith iteration process on the type Scale, to determine the weight coefficient of the video feature extraction network on this type, wherein the correct scale is negatively correlated with the weight coefficient.

The number of correct texts refers to the number of correct text training results generated during the model training process, for example, the number of text training results whose correct probability reaches a probability threshold. The total number of texts refers to the total number of textual training results generated during model training.

In some embodiments, the computer device determines the correct amount of text on the genre in the ith iterative process, the total text on the genre in the ith iterative process, and the following weight coefficient formula (2). The weight coefficient of the video feature extraction network on this type.

W=1-(correct/total) (2)

In the formula, W represents the weight coefficient of the video feature extraction network on this type; correct represents the correct text quantity on this type; total represents the total text quantity on this type.

In the above embodiment, for each type involved in text generation, the weight coefficient corresponding to each type is determined according to the correct ratio corresponding to each type, since the correct ratio represents the ratio of the correct text quantity to the total text quantity, and since the correct ratio represents the ratio of the correct text quantity to the total text quantity There is a negative correlation between the correct scale and the weight coefficient. Therefore, in the case of calculating the text generation loss value, set a smaller weight coefficient for the type with a large correct scale, and set a larger weight coefficient for the type with a small correct scale. In order to determine the accuracy of the weight coefficient, the accuracy of determining the text generation loss value is also improved.

For the i-th iteration process, in some embodiments, based on the above CEloss formula (1), the cross entropy loss values corresponding to the multiple types are calculated, and based on the above weight coefficient formula (2), the multiple types are calculated and obtained After the corresponding weight coefficients, based on the cross-entropy loss values corresponding to the multiple types, the weight coefficients corresponding to the multiple types, and the following loss value formula (3), a weighted sum is performed to obtain the text of the i-th iteration process. Generate loss values.

In the formula, loss _{text generation} represents the loss value of text generation; n represents the number of multiple types; W _s represents the weight coefficient corresponding to type s; CEloss _s represents the cross entropy loss value corresponding to type s.

In the above embodiment, for each type involved in text generation, a weight coefficient corresponding to each type is respectively set, and then the cross-entropy loss value on each type and the weight coefficient corresponding to each type are used to determine the text generation loss value, Improved the accuracy of determining text generation loss values.

In step 408C, the computer device determines the model loss value of the ith iteration process based on the image reconstruction loss value and the text generation loss value.

In some embodiments, a weighted sum is performed based on the image reconstruction loss value, the weight coefficient corresponding to the image reconstruction loss value, the text generation loss value, and the weight coefficient corresponding to the text generation loss value to obtain the ith iteration The model loss value for the process.

In some embodiments, weighting is performed based on the image reconstruction loss value, the weight coefficient corresponding to the image reconstruction loss value, the text generation loss value, the weight coefficient corresponding to the text generation loss value, and the following loss value formula (4). Sum up to get the model loss value of the ith iteration process.

Totalloss=W _{image reconstruction} *loss _{image reconstruction} +W _{text generation} *loss _{text generation} (4)

In the formula, Totalloss represents the model loss value; W _{image reconstruction} represents the weight coefficient corresponding to the image reconstruction loss value; loss _{image reconstruction} represents the image reconstruction loss value; W _{text generation} represents the weight coefficient corresponding to the text generation loss value; loss _{text generation} represents the text generation loss value.

In the embodiment of the present disclosure, for the image reconstruction task and the text generation task of the video feature extraction model, weight coefficients corresponding to each task are respectively set, and then the loss value of each task and the weight coefficient corresponding to each task are used to determine the model loss value , which improves the accuracy of determining the loss value of the model.

In the above embodiment, in any iterative process of model training, the model loss value of this iterative process is used to adjust the model parameters of each sub-model in the video feature extraction model, so as to improve the performance of the video feature extraction model. Text generation ability, so as to train a video feature extraction model with high text generation ability.

In the above embodiment, by constructing the image feature extraction sub-model in the video feature extraction model, the image features of the sample video can be accurately extracted, and by constructing the feature fusion sub-model in the video feature extraction model, not only the sample can be obtained The text features of the video can also feature fusion of the image features and text features of the sample video, so that based on the fusion features, on the one hand, the sample video can be reconstructed to obtain high-quality image features. The content description text of the sample video can be generated, a training method of a video feature extraction model combining an image reconstruction task and a text generation task is provided, and the training effect of the video feature extraction model is improved.

In the technical solution provided by the embodiments of the present disclosure, the image information and text information of the sample video and the text label and image label of the sample video are used to perform model training on the video feature extraction model. The feature extraction sub-model can accurately extract the image features of the sample video. By constructing the feature fusion sub-model in the video feature extraction model, not only the text features of the sample video can be obtained, but also the image features and text of the sample video. Feature fusion is performed on features, so that based on the fusion features, on the one hand, image reconstruction can be performed on the sample video to obtain high-quality image features, and on the other hand, the content description text of the sample video can also be generated. A model training method based on dual training tasks. In the case where the text generation task is the main task and the image reconstruction task is the auxiliary task, since the image label of the sample video represents the image reconstruction feature, in the process of model training, it can be Improve the ability of the video feature extraction model to extract image features, and then obtain high-quality image features. On the basis of obtaining high-quality image features, a video feature extraction model with better text generation ability can be trained to improve The training effect of the video feature extraction model.

In the solution shown in FIG. 4 above, a training method for a video feature extraction model is provided. In some embodiments, a video feature extraction model based on the video feature extraction model trained by the above training method can be implemented. Model text generation method, FIG. 6 is a flow chart of a text generation method based on a video feature extraction model shown according to an exemplary embodiment, referring to FIG. 6, the method includes:

In step 601, the computer device acquires image information and text information of the target video.

In the embodiment of the present disclosure, the target video is used to refer to the video to be subjected to text generation. In some embodiments, taking a computer device provided as a terminal as an example, the target video is a video stored locally by the terminal, or a video downloaded by the terminal, etc.; in other embodiments, taking a computer device provided as a server as an example, The target video is a video in a video database associated with the server, or a video uploaded by the terminal. This embodiment of the present disclosure does not limit the source of the target video.

In some embodiments, the process of acquiring the image information of the target video by the computer device includes at least one of the following: acquiring a cover image of the target video; or, acquiring at least one frame of image in the target video. It should be noted that, for the process of acquiring the image information of the target video, refer to the process of acquiring the image information of the sample video in step 401, which will not be repeated.

In some embodiments, the process of obtaining the text information of the target video by the computer device includes at least one of the following: obtaining description information of the target video; obtaining title information of the target video; obtaining subtitle information of the target video; obtaining the target video The text recognition result of the video; obtain the audio recognition result of the target video. It should be noted that, for the process of acquiring the text information of the target video, refer to the process of acquiring the text information of the sample video in step 401, and details are not repeated here.

In step 602, the computer device inputs the image information and the text information into the video feature extraction model, and performs feature extraction on the image information through an image feature extraction sub-model of the video feature extraction model to obtain image features of the target video.

It should be noted that, for the process of acquiring the image feature of the target video, refer to the process of acquiring the image feature of the sample video in step 403, which will not be repeated.

In step 603, the text information is processed through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain text features of the target video.

It should be noted that, for the process of acquiring the text feature of the target video, refer to the process of acquiring the text feature of the sample video in step 404, and details are not repeated here.

In step 604, feature fusion is performed on the image feature and the text feature through the feature fusion layer of the feature fusion sub-model to obtain the fusion feature of the target video.

It should be noted that, for the process of acquiring the fusion feature of the target video, refer to the process of acquiring the fusion feature of the sample video in step 405, and details are not repeated here.

In step 605, the fusion feature is processed by the text generation sub-model of the video feature extraction model, a plurality of characters satisfying the text generation condition are output, and a content description text of the target video is generated based on the plurality of characters.

It should be noted that, for the process of acquiring the content description text of the target video, refer to the process of acquiring the text training result of the sample video in step 407, and details are not repeated here.

In the above embodiment, the computer device inputs the image information and the text information into the video feature extraction model, processes the image information and the text information through the video feature extraction model, and outputs a plurality of characters that meet the text generation conditions, The content description text of the target video is generated based on the plurality of characters, and a model for text generation based on image information and text information is provided. With reference to the multi-modal information of the image modality and the text model, video feature extraction is added. The amount of information referenced by the model, on the one hand, enriches the content description text generated by the video feature extraction model, and on the other hand, improves the accuracy of text generation by the video feature extraction model. The modality refers to the representation or representation of information, and it should be understood that each medium or form of information can be referred to as a modality, for example, a medium of information such as images, texts, audios, and so on. In other embodiments, the computer device can also use information of other modalities to perform the above text generation process, for example, the release information of the target video and so on.

In some embodiments, the method further includes: performing image restoration on the image features in the fusion feature by using the image reconstruction sub-model of the video feature extraction model to obtain the image reconstruction feature of the original image size of the target video. It should be noted that, for the process of acquiring the image reconstruction feature of the target video, refer to the process of acquiring the image training result of the sample video in step 406, and details are not repeated here.

According to the technical solutions provided by the embodiments of the present disclosure, by constructing an image feature extraction sub-model in the video feature extraction model, the image features of the target video can be accurately extracted, and by constructing a feature fusion sub-model in the video feature extraction model, not only can To the text features of the target video, it can also perform feature fusion on the image features and text features of the target video, so as to follow the fusion features, on the one hand, the target video can be reconstructed to obtain high-quality image features, on the other hand In one aspect, the content description text of the target video can also be generated, and a video feature extraction model combining image reconstruction task and text generation task is provided, which can better characterize the target video and improve the accuracy of video representation.

Fig. 7 is a block diagram of an apparatus for training a video feature extraction model according to an exemplary embodiment. Referring to FIG. 7 , the apparatus includes an acquisition unit 701 , an input unit 702 , a processing unit 703 and a training unit 704 .

an acquisition unit 701, configured to perform acquisition of image information, text information, image tags and text tags of the sample video, where the image tag represents an image reconstruction feature, and the text tag represents the content description text of the sample video;

The input unit 702 is configured to input the image information and the text information into the video feature extraction model, and perform feature extraction on the image information through the image feature extraction sub-model of the video feature extraction model to obtain the image feature of the sample video, The text information is processed through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain the text features of the sample video, and the image features and the text features are feature-fused through the feature fusion layer of the feature fusion sub-model. , to obtain the fusion feature of the sample video;

The processing unit 703 is configured to perform image restoration through the image reconstruction sub-model of the video feature extraction model to the image features in the fusion feature to obtain an image training result of the original image size, and the text generation sub-model of the video feature extraction model is used. The model processes the fusion feature to obtain the text training result;

The training unit 704 is configured to perform image labeling and text labeling based on the image training result, the text training result and the sample video, and adjust the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model, and the image reconstruction sub-model. Model parameters for the text generation sub-model to train this video feature extraction model.

In the technical solution provided by the embodiments of the present disclosure, the image information and text information of the sample video and the text label and image label of the sample video are used to perform model training on the video feature extraction model. The feature extraction sub-model can accurately extract the image features of the sample video. By constructing the feature fusion sub-model in the video feature extraction model, not only the text features of the sample video can be obtained, but also the image features and text of the sample video. Feature fusion is performed on features, so that based on the fusion features, on the one hand, image reconstruction can be performed on the sample video to obtain high-quality image features, and on the other hand, the content description text of the sample video can also be generated. A model training method based on dual training tasks. In the case where the text generation task is the main task and the image reconstruction task is the auxiliary task, since the image label of the sample video represents the image reconstruction feature, in the process of model training, it can be Improve the ability of the video feature extraction model to extract image features, and then obtain high-quality image features. On the basis of obtaining high-quality image features, a video feature extraction model with better text generation ability can be trained to improve The training effect of the video feature extraction model.

In some embodiments, the obtaining unit 701 is configured to perform at least one of the following:

Obtain the cover image of the sample video; or, obtain at least one frame of image in the sample video.

In some embodiments, the obtaining unit 701 is configured to perform at least one of the following:

Obtain the description information of the sample video; obtain the title information of the sample video; obtain the subtitle information of the sample video; Identify the obtained result; obtain the audio recognition result of the sample video, where the audio recognition result is the result obtained by performing audio recognition on the background audio of the sample video.

In some embodiments, the content description text is at least one of content category description text, content form description text, content topic description text, and content detail description text.

In some embodiments, the input unit 702 includes a processing subunit configured to perform any of the following:

Through the self-attention layer included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video;

Through the deep belief network included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video.

In some embodiments, the processing unit 703 includes a text generation sub-unit configured to perform:

Through the self-attention layer included in the text generation sub-model, the fusion feature is processed to obtain the text training result.

In some embodiments, the text training result includes multiple types of content description text;

The device also includes an adding unit configured to perform on the fusion feature, adding type identifiers of various types;

The processing unit 703 includes a text generation sub-unit, and is further configured to input the fusion feature after adding the type identification into the text generation sub-model, and through the text generation sub-model, based on the corresponding processing mechanisms of each type identification, the The fusion features are processed to obtain the multiple types of content description texts.

In some embodiments, the number of the text information is multiple;

The device also includes a splicing unit configured to perform splicing of a plurality of the text information to obtain the spliced text information;

The input unit is further configured to perform the step of processing the text information through the embedding layer of the feature fusion sub-model of the video feature extraction model based on the spliced text information to obtain text features of the sample video.

In some embodiments, the input unit is further configured to perform:

From the spliced text information, extract the characters of the previous target number;

Based on the extracted characters, the step of processing the text information through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain text features of the sample video is performed.

In some embodiments, the training unit 704 includes:

The determining subunit is configured to perform in the ith iteration process of model training, based on the image training result of the ith iteration process, the text training result and the image label and text label of the sample video, determine the ith time The model loss value of the iterative process, the i is a positive integer greater than 1;

The adjustment subunit is configured to execute the model loss value based on the i-th iteration process, adjust the model parameters of the video feature extraction model determined in the i-1-th iteration process, and repeat the above-mentioned iterative process of training until training meet the target conditions.

In some embodiments, the determining subunit includes:

The image reconstruction loss value determination subunit is configured to execute the image training result based on the i-th iteration process and the image label of the sample video, and determine the image reconstruction loss value of the i-th iteration process, and the image reconstruction loss value represents the difference between the training result of the image and the label of the image;

The text generation loss value determination subunit is configured to execute the text generation loss value of the ith iteration process based on the text training result of the ith iteration process and the text label of the sample video, and the text generation loss value represents the difference between the text training result and the text label;

The model loss value determination subunit is configured to perform the reconstruction loss value based on the image and the text generation loss value, and determine the model loss value of the ith iteration process.

In some embodiments, the model loss value determination subunit is configured to perform:

Based on the image reconstruction loss value, the weight coefficient corresponding to the image reconstruction loss value, the text generation loss value, and the weight coefficient corresponding to the text generation loss value, a weighted sum is performed to obtain the model loss value of the i-th iteration process.

In some embodiments, the text training result includes multiple types of content description text;

The text generation loss value determination subunit is configured to perform:

For any type, based on the text training result of the i-th iteration process on the type and the description text label of the sample video on the type, determine the loss value of the i-th iteration process on the type;

Based on the cross-entropy loss value of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, weighted summation is performed to obtain the text generation loss of the i-th iteration process value.

In some embodiments, the apparatus further includes a determining unit configured to perform:

For any type, based on the correct number of texts on the type and the total number of texts in the ith iteration process, determine the correct scale of the ith iteration process on the type, the correct scale representing the ith iteration process The ratio of the number of correct texts to the total number of texts in the process;

Based on the correct scale of the i-th iterative process on the type, the weight coefficient of the video feature extraction network on the type is determined, wherein the correct scale is negatively correlated with the weight coefficient.

Fig. 8 is a block diagram of a text generation apparatus based on a video feature extraction model according to an exemplary embodiment. Referring to FIG. 8 , the apparatus includes an acquisition unit 801 , an input unit 802 and a processing unit 803 .

an acquisition unit 801, configured to perform acquisition of image information and text information of the target video;

The input unit 802 is configured to input the image information and the text information into the video feature extraction model, and perform feature extraction on the image information through the image feature extraction sub-model of the video feature extraction model to obtain the image feature of the target video. , the text information is processed through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain the text features of the target video, and the image features and the text features are characterized by the feature fusion layer of the feature fusion sub-model. Fusion to obtain the fusion feature of the target video;

The processing unit 803 is configured to process the fusion feature through the text generation sub-model of the video feature extraction model, output a plurality of characters satisfying the text generation condition, and generate a content description text of the target video based on the plurality of characters.

According to the technical solutions provided by the embodiments of the present disclosure, by constructing an image feature extraction sub-model in the video feature extraction model, the image features of the target video can be accurately extracted, and by constructing a feature fusion sub-model in the video feature extraction model, not only can To the text features of the target video, it can also perform feature fusion on the image features and text features of the target video, so that the subsequent processing based on the fusion features can output multiple characters that meet the text generation conditions, and then based on the output multiple characters The content description text of the target video can be automatically generated, and a video feature extraction model based on text generation is provided. The generated content description text contains a wealth of information, which can better characterize the target video and improve the video representation. accuracy.

In some embodiments, the processing unit 803 is further configured to execute:

Image reconstruction is performed on the image features in the fusion feature through the image reconstruction sub-model of the video feature extraction model to obtain the image reconstruction feature of the original image size of the target video.

It should be noted that: the training device for the video feature extraction model provided in the above-mentioned embodiments only uses the division of the above-mentioned functional modules as an example for the feature extraction. In practical applications, the above-mentioned functions may be allocated to different functions as required Module completion means dividing the internal structure of the device into different functional modules to complete all or part of the functions described above. In addition, the apparatus for training a video feature extraction model provided in the above embodiment and the embodiment of the training method for a video feature extraction model belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

The computer device mentioned in the embodiments of the present disclosure may be provided as a terminal. FIG. 9 shows a structural block diagram of a terminal 900 provided by an exemplary embodiment of the present disclosure. The terminal 900 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, the standard audio level 3 of Moving Picture Experts compression), MP4 (Moving Picture Experts Group Audio Layer IV, the standard audio of Moving Picture Experts Group compression) Level 4) Player, laptop or desktop computer. Terminal 900 may also be called user equipment, portable terminal, laptop terminal, desktop terminal, and the like by other names.

Generally, the terminal 900 includes: a processor 901 and a memory 902 .

The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 901 may use at least one hardware form of DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), and PLA (Programmable Logic Array, programmable logic array). accomplish. The processor 901 may also include a main processor and a co-processor. The main processor is a processor used to process data in a wake-up state, also called a CPU (Central Processing Unit, central processing unit); the co-processor is a A low-power processor for processing data in a standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor 901 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is used to process computing operations related to machine learning.

Memory 902 may include one or more computer-readable storage media, which may be non-transitory. Memory 902 may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, a non-transitory computer-readable storage medium in the memory 902 is used to store at least one program code, and the at least one program code is used to be executed by the processor 901 to implement the methods provided by the method embodiments of the present disclosure. The process performed by the terminal in the training method of the video feature extraction model or the text generation method based on the video feature extraction model.

In some embodiments, the terminal 900 may optionally further include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 may be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 903 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904 , a display screen 905 , a camera assembly 906 , an audio circuit 907 , a positioning assembly 908 and a power supply 909 .

The peripheral device interface 903 may be used to connect at least one peripheral device related to I/O (Input/Output) to the processor 901 and the memory 902 . In some embodiments, processor 901, memory 902, and peripherals interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one of processor 901, memory 902, and peripherals interface 903 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The radio frequency circuit 904 is used for receiving and transmitting RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals. The radio frequency circuit 904 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 904 may communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: metropolitan area network, mobile communication networks of various generations (2G, 3G, 4G and 5G), wireless local area network and/or WiFi (Wireless Fidelity, wireless fidelity) network. In some embodiments, the radio frequency circuit 904 may further include a circuit related to NFC (Near Field Communication, short-range wireless communication), which is not limited in the present disclosure.

The display screen 905 is used to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 also has the ability to acquire touch signals on or above the surface of the display screen 905 . The touch signal may be input to the processor 901 as a control signal for processing. At this time, the display screen 905 may also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards. In some embodiments, there may be one display screen 905, which is arranged on the front panel of the terminal 900; in other embodiments, there may be at least two display screens 905, which are respectively arranged on different surfaces of the terminal 900 or in a folded design; In other embodiments, the display screen 905 may be a flexible display screen, which is disposed on a curved surface or a folding surface of the terminal 900 . Even, the display screen 905 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light emitting diode).

The camera assembly 906 is used to capture images or video. Optionally, the camera assembly 906 includes a front camera and a rear camera. Usually, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth-of-field camera to realize the background blur function, the main camera It is integrated with the wide-angle camera to achieve panoramic shooting and VR (Virtual Reality, virtual reality) shooting functions or other integrated shooting functions. In some embodiments, camera assembly 906 may also include a flash. The flash can be a single color temperature flash or a dual color temperature flash. Dual color temperature flash refers to the combination of warm light flash and cold light flash, which can be used for light compensation under different color temperatures.

Audio circuitry 907 may include a microphone and speakers. The microphone is used to collect the sound waves of the user and the environment, convert the sound waves into electrical signals, and input them to the processor 901 for processing, or to the radio frequency circuit 904 to realize voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively disposed in different parts of the terminal 900 . The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 901 or the radio frequency circuit 904 into sound waves. The loudspeaker can be a traditional thin-film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for distance measurement and other purposes. In some embodiments, the audio circuit 907 may also include a headphone jack.

The positioning component 908 is used to locate the current geographic location of the terminal 900 to implement navigation or LBS (Location Based Service, location-based service).

The power supply 909 is used to power various components in the terminal 900 . The power source 909 may be alternating current, direct current, primary batteries, or rechargeable batteries. When the power source 909 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, terminal 900 also includes one or more sensors 910 . The one or more sensors 910 include, but are not limited to, an acceleration sensor 911 , a gyro sensor 912 , a pressure sensor 913 , a fingerprint sensor 914 , an optical sensor 915 , and a proximity sensor 916 .

The acceleration sensor 911 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 900 . For example, the acceleration sensor 911 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 901 may control the display screen 905 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 911 . The acceleration sensor 911 can also be used for game or user movement data collection.

The gyroscope sensor 912 can detect the body direction and rotation angle of the terminal 900 , and the gyroscope sensor 912 can cooperate with the acceleration sensor 911 to collect 3D actions of the user on the terminal 900 . The processor 901 can implement the following functions according to the data collected by the gyro sensor 912 : motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

The pressure sensor 913 may be disposed on the side frame of the terminal 900 and/or the lower layer of the display screen 905 . When the pressure sensor 913 is disposed on the side frame of the terminal 900, the user's holding signal of the terminal 900 can be detected, and the processor 901 can perform left and right hand identification or shortcut operations according to the holding signal collected by the pressure sensor 913. When the pressure sensor 913 is disposed on the lower layer of the display screen 905, the processor 901 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 905. The operability controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

Optical sensor 914 is used to collect ambient light intensity. In one embodiment, the processor 901 may control the display brightness of the display screen 905 according to the ambient light intensity collected by the optical sensor 914 . Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 may also dynamically adjust the shooting parameters of the camera assembly 906 according to the ambient light intensity collected by the optical sensor 914 .

A proximity sensor 915 , also called a distance sensor, is usually provided on the front panel of the terminal 900 . The proximity sensor 915 is used to collect the distance between the user and the front of the terminal 900 . In one embodiment, when the proximity sensor 915 detects that the distance between the user and the front of the terminal 900 gradually decreases, the processor 901 controls the display screen 905 to switch from the bright screen state to the off screen state; when the proximity sensor 915 detects When the distance between the user and the front of the terminal 900 gradually increases, the processor 901 controls the display screen 905 to switch from the closed screen state to the bright screen state.

Those skilled in the art can understand that the structure shown in FIG. 9 does not constitute a limitation on the terminal 900, and may include more or less components than shown, or combine some components, or adopt different component arrangements.

The computer device mentioned in the embodiments of the present disclosure may be provided as a kind of server. FIG. 10 is a block diagram of a server according to an exemplary embodiment. The server 1000 may vary greatly due to different configurations or performance, and may include one or more processors (Central Processing Units, CPU) 1001 and One or more memories 1002, wherein, at least one piece of program code is stored in the one or more memories 1002, and the at least one piece of program code is loaded and executed by the one or more processors 1001 to realize the above-mentioned various method embodiments provided The process performed by the server in the training method of the video feature extraction model. Of course, the server 1000 may also have components such as wired or wireless network interfaces, keyboards, and input/output interfaces for input and output, and the server 1000 may also include other components for implementing device functions, which will not be repeated here.

In an exemplary embodiment, a computer-readable storage medium including program codes is also provided, such as a memory 1002 including program codes, and the program codes can be executed by the processor 1001 of the server 1000 to complete the training of the above-mentioned video feature extraction model method. Optionally, the computer-readable storage medium may be ROM (Read-Only Memory, read-only memory), RAM (Random Access Memory, random access memory), CD-ROM (Compact-Disc Read-Only Memory, CD-ROM) ), magnetic tapes, floppy disks, and optical data storage devices, etc.

In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-mentioned training method for a video feature extraction model.

In some embodiments, the computer programs involved in the embodiments of the present disclosure may be deployed and executed on one computer device, or executed on multiple computer devices located at one site, or distributed over multiple sites and through communication Executed on multiple computer devices interconnected by a network, and multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common general knowledge or techniques in the technical field not disclosed by this disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. a training method of video feature extraction model, is characterized in that, described method comprises: Obtain image information, text information, image tags and text tags of the sample video, where the image tag represents an image reconstruction feature, and the text tag represents the content description text of the sample video; Input the image information and the text information into the video feature extraction model, and perform feature extraction on the image information through the image feature extraction sub-model of the video feature extraction model to obtain the image features of the sample video, and obtain the image features of the sample video. The embedded layer of the feature fusion sub-model of the video feature extraction model processes the text information to obtain the text features of the sample video, and the image features and the text features are analyzed by the feature fusion layer of the feature fusion sub-model. Perform feature fusion to obtain fusion features of the sample video; Image restoration is performed on the image features in the fusion features through the image reconstruction sub-model of the video feature extraction model to obtain an image training result of the original image size. Features are processed to obtain text training results; Adjust the image feature extraction sub-model, the feature fusion sub-model, the image reconstruction sub-model and the text based on the image training result, the text training result and the image label and text label of the sample video Model parameters of the sub-model are generated to train the video feature extraction model. 2. The training method of the video feature extraction model according to claim 1, wherein the content description text is at least one of content category description text, content form description text, content subject description text and content detail description text. one type. 3. The training method of the video feature extraction model according to claim 1, wherein the feature fusion layer of the feature fusion sub-model performs feature fusion on the image feature and the text feature to obtain the The fusion features of the sample video include any of the following: Through the self-attention layer included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video; Through the deep belief network included in the feature fusion sub-model, the image feature and the text feature are processed to obtain the fusion feature of the sample video. 4. the training method of the video feature extraction model according to claim 1, is characterized in that, described fusion feature is processed by the text generation sub-model of described video feature extraction model, and obtaining text training result comprises: Through the self-attention layer included in the text generation sub-model, the fusion feature is processed to obtain the text training result. 5. A text generation method based on a video feature extraction model, wherein the video feature extraction model is trained based on the training method described in any one of the above claims 1 to 4, and the method comprises: Obtain the image information and text information of the target video; The image information and the text information are input into the video feature extraction model, and feature extraction is performed on the image information through the image feature extraction sub-model of the video feature extraction model to obtain the image features of the target video. The embedding layer of the feature fusion sub-model of the video feature extraction model processes the text information to obtain the text features of the target video, and the image features are compared with the image features through the feature fusion layer of the feature fusion sub-model. Perform feature fusion on text features to obtain fusion features of the target video; The fusion feature is processed by the text generation sub-model of the video feature extraction model, a plurality of characters satisfying text generation conditions are output, and a content description text of the target video is generated based on the plurality of characters. 6. A training device for a video feature extraction model, wherein the device comprises: an acquisition unit, configured to perform acquisition of image information, text information, image tags and text tags of the sample video, where the image tag represents an image reconstruction feature, and the text tag represents the content description text of the sample video; an input unit, configured to input the image information and the text information into a video feature extraction model, and perform feature extraction on the image information through an image feature extraction sub-model of the video feature extraction model to obtain the sample video The image features of the sample video are processed through the embedding layer of the feature fusion sub-model of the video feature extraction model to obtain the text features of the sample video, and the feature fusion layer of the feature fusion sub-model is used to process the text information. The image feature and the text feature are feature-fused to obtain the fusion feature of the sample video; A processing unit, configured to perform image restoration on the image features in the fusion features through the image reconstruction sub-model of the video feature extraction model, to obtain an image training result of the original image size, and through the text of the video feature extraction model generating a sub-model to process the fusion feature to obtain a text training result; A training unit configured to adjust the image feature extraction sub-model, the feature fusion sub-model, the image The reconstruction sub-model and the model parameters of the text generation sub-model are used to train the video feature extraction model. 7. A text generation device based on a video feature extraction model, wherein the video feature extraction model is trained based on the training method described in any one of claims 1 to 4, and the device comprises: an acquisition unit, configured to perform acquisition of image information and text information of the target video; an input unit, configured to input the image information and the text information into the video feature extraction model, and perform feature extraction on the image information through an image feature extraction sub-model of the video feature extraction model to obtain the The image features of the target video are processed through the embedding layer of the feature fusion sub-model of the video feature extraction model to process the text information, and the text features of the target video are obtained. Feature fusion is performed on the image feature and the text feature to obtain the fusion feature of the target video; a processing unit, configured to process the fusion feature through a text generation sub-model of the video feature extraction model, output a plurality of characters satisfying text generation conditions, and generate the content of the target video based on the plurality of characters Description text. 8. A computer device, characterized in that the computer device comprises: one or more processors; memory for storing said processor executable program code; Wherein, the processor is configured to execute the program code, so as to realize the training method of the video feature extraction model according to any one of claims 1 to 4, or the video feature extraction model based on the video feature extraction model according to claim 5 text generation method. 9. A computer-readable storage medium, characterized in that, when the program code in the computer-readable storage medium is executed by a processor of a computer device, the computer device is enabled to execute any one of claims 1 to 4 The training method of the video feature extraction model, or the text generation method based on the video feature extraction model of claim 5. 10. A computer program product, comprising a computer program, characterized in that, when the computer program is executed by a processor, the training method of the video feature extraction model according to any one of claims 1 to 4 is realized, or claim 5 The text generation method based on the video feature extraction model.

Images