CN115116475A - A method and device for automatic detection of speech depression based on time-delay neural network - Google Patents

Abstract

The invention provides a voice depression automatic detection method and a device based on a time delay neural network, wherein the method comprises the steps of obtaining an initial voice signal, dividing the initial voice signal into a plurality of voice sections, each voice section comprises at least one voice frame, and respectively calculating the short-time energy and the short-time zero-crossing rate of each voice section in the initial voice signal; obtaining effective voice fragments based on the short-time energy and the short-time zero crossing rate; carrying out pre-emphasis processing on each effective voice fragment, framing the pre-emphasized effective voice fragments based on time to obtain a plurality of frame fragments, and calculating a Mel frequency cepstrum coefficient corresponding to each frame fragment; inputting Mel frequency cepstrum coefficients into a preset time delay neural network model, extracting frame level characteristics by adopting a hierarchical residual convolution and compression excitation mechanism, merging the frame level characteristics based on statistics pooling of an attention mechanism, and obtaining probability parameters through a classification model; and finally, voting and integrating to obtain a prediction result.

Description

A method and device for automatic detection of speech depression based on time-delay neural network

technical field

The invention relates to the technical field of speech processing, in particular to a method and device for automatic detection of speech depression based on a time-delay neural network.

Background technique

Depression is a common mental illness, mainly manifested as low mood, slow thinking and decreased willpower, and has become one of the major health problems worldwide. Another factor that causes the serious harm of depression is the lack of objective examination methods for the diagnosis of depression. For its evaluation and diagnosis, it mainly depends on the psychiatric examination of the neurologist, which largely depends on the subjective experience of the doctor, and the diagnostic tools. Also limited to questionnaires and diagnostic scales.

Existing methods for diagnosing depression mainly rely on the diagnosis experience of doctors, so they have high requirements on the experience of doctors, and it is difficult for doctors with less experience to ensure the quality of diagnosis.

As the most direct way for humans to transmit information, speech contains a lot of information about human health status. A large number of studies have shown that the pronunciation characteristics of patients with depression are significantly different from those of normal people, such as fundamental frequency, loudness, and speech rate. characteristics will vary greatly.

Therefore, there is an urgent need for a depression diagnosis method based on artificial intelligence and speech signal processing technology in the prior art.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present invention provide a method and device for automatic detection of speech depression based on a time-delay neural network, so as to eliminate or improve one or more defects existing in the prior art.

A first aspect of the present invention provides an automatic detection method for speech depression based on a time-delay neural network, and the steps of the method include:

Obtain an initial speech signal, divide the initial speech signal into multiple speech segments, each speech segment includes at least one speech frame, and separately calculate the short-term energy and short-term zero-crossing rate of each speech segment in the initial speech signal ;

Acquire the voiced segment in the speech segment of the initial speech signal based on the short-term energy, acquire the unvoiced segment in the speech segment of the initial speech signal based on the short-term zero-crossing rate, and combine all voiced segments in the initial speech signal Fragments and unvoiced fragments to obtain valid voice fragments;

Pre-emphasis is performed on each of the effective speech segments, and the pre-emphasized effective speech segments are divided into frames based on time to obtain multiple frame segments, and the Mel frequency cepstral coefficients corresponding to each frame segment are calculated;

Inputting the Mel-frequency cepstral coefficients into a preset time-delay neural network model, and calculating a feature vector corresponding to the Mel-frequency cepstral coefficients based on the feature extraction module of the time-delay neural network model, based on the time delay The feature aggregation module of the neural network model calculates the mean and variance corresponding to each feature vector, and inputs the mean and variance corresponding to each feature vector into the classification module of the time-delay neural network model to obtain probability parameters.

By adopting the above scheme, compared with the method of using the depression scale for diagnosis, the present invention does not need to rely on the experience of professional doctors for diagnosis, and does not require expensive infrastructure and complicated operation procedures. The present invention is based on the Mel frequency cepstral coefficient ( MFCC) feature extraction of speech features, using deep learning methods to process, segmenting long segments of speech, as time-delay neural network input classification results, and integrating to obtain depression diagnosis results.

In some embodiments of the present invention, before the step of inputting the Mel-frequency cepstral coefficients into the preset time-delay neural network model, the step further includes,

The feature data enhancement is performed on the Mel-frequency cepstral coefficients through a spectral mask, and the enhanced Mel-frequency cepstral coefficients are input into a preset time-delay neural network model.

In some embodiments of the present invention, the manner of spectral masking includes, but is not limited to, time-domain masking or frequency-domain masking.

In some embodiments of the present invention, in the step of separately calculating the short-term energy and short-term zero-crossing rate of each speech segment in the initial speech signal, the short-term energy is calculated based on the following formula:

Ex represents the short-term energy of speech segment _x , N represents the total number of frames in speech segment x, n represents any one of N frames, and x[n] represents the amplitude of the nth frame in N frames.

In some embodiments of the present invention, in the step of separately calculating the short-term energy and short-term zero-crossing rate of each speech segment in the initial speech signal, the short-term zero-crossing rate is calculated based on the following formula:

Z _x represents the short-term zero-crossing rate of the speech segment x, N represents the total number of frames in the speech segment x, n represents any one of the N frames, x(n) represents the amplitude of the nth frame in the N frames, x(n-1) represents the magnitude of the n-1th frame in N frames, and sgn represents the sign function.

In some embodiments of the present invention, when the voiced segment in the speech segment of the initial speech signal is acquired based on the short-term energy, the unvoiced segment in the speech segment of the initial speech signal is acquired based on the short-term zero-crossing rate Fragment steps,

Preset short-term energy threshold and short-term zero-crossing rate threshold;

Based on comparing the short-term energy value and the short-term energy threshold of each speech segment, the voiced segments in the speech segment are obtained;

Based on comparing the short-term zero-crossing rate value and the short-term zero-crossing rate threshold of each speech segment, the unvoiced segments in the speech segment are obtained.

In some embodiments of the present invention, pre-emphasis processing is performed on each of the valid speech segments based on the following formula;

y(n)=x(n)-αx(n-1)

x(n) represents the amplitude of the nth frame in the N frames, x(n-1) represents the amplitude of the n-1th frame in the N frames, and y(n) is the amplitude of the valid speech segment after pre-emphasis processing. The amplitude of the nth frame among the N frames, and α is the pre-emphasis factor.

In some embodiments of the present invention, in the step of dividing the pre-emphasized effective speech segment into frames based on time to obtain multiple frame segments,

Each valid speech segment of the first time length is divided into a frame segment, and adjacent frame segments have overlapping segments of the second time length.

In some embodiments of the present invention, the step of calculating the Mel frequency cepstral coefficient corresponding to each frame segment includes:

Windowing is performed on each frame segment based on the window function;

Perform fast Fourier transform on the windowed frame segment to convert the time domain signal into a frequency domain signal;

Convert the frequency of the frequency domain signal to the Mel frequency based on the Mel filter to obtain the Mel frequency signal;

Perform inverse Fourier transform on the Mel-frequency signal, convert the Mel-frequency signal to the time domain, and obtain the Mel-frequency cepstral coefficient.

In some embodiments of the present invention, the feature extraction module includes a plurality of consecutive Se-Res2 modules, each Se-Res2 module is provided with a Res2Net layer for convolution processing, and the feature extraction module adopts hierarchical residual connection Extracting frame-level features; the feature aggregation module includes an attention mechanism layer, and calculates the mean and variance corresponding to each feature vector based on the attention mechanism; the classification module includes sequentially connected fully connected layers and Softmax layers, which are output by the Softmax layer probability parameter.

In some embodiments of the present invention, in the convolution processing step of the Res2Net layer, the mode of hierarchical residual connection is introduced, and the features are split on the channel during the one-dimensional hole convolution, and the depression-related features of different scales are carried out. Extract and then fuse the grouped features. Embedded compression excitation module, which uses global information to evaluate the importance of each feature channel, that is, learns the weight information that characterizes the importance of each channel, readjusts the features of each channel output after convolution, and realizes prominence is more critical for depression diagnosis information and suppress irrelevant redundant information.

In some embodiments of the present invention, in the step of extracting frame-level features by using hierarchical residual connections in the feature extraction module:

The Mel frequency cepstral coefficients are adjusted into the first feature map through a one-dimensional convolution, and the first feature map is input into the Se-Res2 module, and the input data is divided into four feature sub-sections on average through the Se-Res2 module each time. The graphs are convolved separately, the convolved feature sub-maps are spliced, the spliced feature maps are again subjected to one-dimensional convolution to obtain the output of the hierarchical residual convolution, and the second feature is output by the last Se-Res2 module. Figure, according to the following formula, convolve the feature subgraph:

y _i represents the feature submap after convolution, i represents the serial number of the feature submap, y _i-1 represents the i-1 th feature submap x _i-1 convolved feature submap, K _i represents the i th feature submap The 3x3 convolution corresponding to the feature submap x _i .

In some embodiments of the present invention, in the step of outputting the second feature map by the last Se-Res2 module, the second feature map output by the last Se-Res2 module is adjusted based on the preset compression excitation module, which specifically includes the steps of ,

The weight factor is obtained based on the preset compression excitation module, and the weight factor is weighted to the output feature map of each Se-Res2 module. The feature map weighted by the weight factor includes the second feature map output by the last Se-Res2 module. After adjustment Second feature map:

According to the following formula, the weight factor is obtained based on the preset compression excitation module:

s=σ ₁ (W ₂ f ₁ (W ₁ z+b ₁ )+b ₂ )

z is the channel descriptor, R represents the total number of frames of the first feature map, r represents the rth frame in the R frame, γ _r represents the feature vector of the rth frame of the first feature map, W ₁ , W ₂ , b ₁ , b ₂ are the parameters of the two fully connected layers, f ₁ is the relu activation function, σ ₁ is the sigmod activation function, and s is the weight factor.

In some embodiments of the present invention, in the step of calculating the mean and variance corresponding to each feature vector based on the attention mechanism:

According to the following formula, the scaling factor corresponding to each eigenvector is calculated and normalized:

e _t =v ^T f ₂ (Wh _t +b)+k;

e _t represents the attention score of the t-th frame segment, f ₂ represents the nonlinear activation function, W represents the weight parameter, h _t represents the feature vector of the t-th frame segment, b represents the bias parameter, v ^T and k are both The preset linear layer learning parameters, α _t represents the attention score normalized by softmax, and T represents the total number of frame segments;

Calculate the mean and variance corresponding to each eigenvector based on the scaling factor according to the following formula:

μ represents the mean, σ ₂ represents the variance, and t represents the t-th frame segment.

A second aspect of the present invention provides a device for automatic detection of speech depression based on a time-delay neural network, the device includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the The processor is configured to execute computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus implements the steps of the above method.

A third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the aforementioned automatic detection method for speech depression based on a time-delay neural network. .

Additional advantages, objects, and features of the present invention will be set forth in part in the description that follows, and in part will become apparent to those of ordinary skill in the art upon study of the following, or may be learned from practice of the invention. The objectives and other advantages of the invention may be particularly pointed out and attained by the description and drawings.

Those skilled in the art will appreciate that the objects and advantages that can be achieved with the present invention are not limited to those specifically described above, and that the above and other objects that can be achieved by the present invention will be more clearly understood from the following detailed description.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention, and constitute a part of the present application, and do not constitute a limitation to the present invention.

1 is a schematic diagram of an embodiment of a method for automatic detection of speech depression based on a time-delay neural network according to the present invention;

Fig. 2 is the overall frame schematic diagram of the automatic detection method of speech depression based on time delay neural network of the present invention;

3 is a schematic flowchart of obtaining Mel frequency cepstral coefficients according to the present invention;

4 is a schematic diagram of processing steps of the time delay neural network model of the present invention;

FIG. 5 is a schematic diagram of a spectrum using a triangular filter method.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

Here, it should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only the structures and/or processing steps closely related to the solution according to the present invention are shown in the drawings, and the related structures and/or processing steps are omitted. Other details not relevant to the invention.

It should be emphasized that the term "comprising/comprising" when used herein refers to the presence of a feature, element, step or component, but does not exclude the presence or addition of one or more other features, elements, steps or components.

Here, it should also be noted that, if there is no special description, the term "connection" herein may not only refer to direct connection, but also to indicate indirect connection with intermediates.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the drawings, the same reference numbers represent the same or similar parts, or the same or similar steps.

In order to solve the above problems, as shown in Figures 1, 2, and 3, the present invention proposes a method for automatic detection of speech depression based on a time-delay neural network. The steps of the method include:

Step S100, obtaining an initial speech signal, dividing the initial speech signal into a plurality of speech segments, each speech segment including at least one speech frame, and calculating the short-term energy and short-term energy of each speech segment in the initial speech signal respectively. zero-crossing rate;

In some embodiments of the present invention, the time length of the speech frame may be 20ms, 30ms, or 50ms, or the like.

Step S200, acquiring voiced segments in the speech segment of the initial speech signal based on the short-term energy, acquiring unvoiced segments in the speech segment of the initial speech signal based on the short-term zero-crossing rate, combining the All voiced segments and unvoiced segments of , get valid speech segments;

In some embodiments of the present invention, the short-term energy represents the average value of the energy of the speech signal, and the short-term zero-crossing rate represents the number of times that the waveform graph of a frame of speech signal crosses the horizontal axis;

Using the above scheme, the speech can be divided into unvoiced, voiced and noise parts, and noise segments need to be removed from them. The short-term energy of voiced sounds is significantly higher than that of unvoiced sounds and noise, and the short-term zero-crossing rate of unvoiced sounds is higher than that of noise. Therefore, The purpose can be achieved by setting an appropriate threshold to accurately remove noise fragments.

Step S300, performing pre-emphasis processing on each of the valid speech segments, dividing the pre-emphasized valid speech segments into frames based on time to obtain multiple frame segments, and calculating the Mel frequency cepstral coefficients corresponding to each frame segment ;

In some embodiments of the present invention, the pre-emphasis process can enhance the high frequency part in the speech signal, so that the frequency spectrum of the signal becomes flat, which is convenient for spectrum or channel parameter analysis.

In some embodiments of the present invention, the time length of the frame segment may be 20ms, 30ms, or 50ms, or the like.

In some embodiments of the present invention, the calculation of the mel-frequency cepstral coefficients corresponding to each frame segment may be implemented by a mel filter.

As shown in FIG. 4 , in step S400, the Mel-frequency cepstral coefficients are input into a preset time-delay neural network model, and a feature extraction module based on the time-delay neural network model calculates and obtains the corresponding Mel-frequency cepstral coefficients The eigenvectors are calculated based on the feature aggregation module of the time-delay neural network model to obtain the mean and variance corresponding to each eigenvector, and the mean and variance corresponding to each eigenvector are input into the classification module of the time-delay neural network model, Get the probability parameter.

In some embodiments of the present invention, the feature extraction module is implemented based on a time-delay neural network, and consists of three consecutive SE-Res2 modules with gradually increasing strides. A time-delay neural network is a neural network architecture that concatenates features from historical, current, and future frames, thereby introducing timing information. The network can be implemented by a multi-layer one-dimensional hole convolutional neural network, which can reduce the overall number of parameters and reduce the amount of calculation.

The Mel frequency cepstral coefficients are adjusted into the first feature map through a one-dimensional convolution, and the first feature map is input into the Se-Res2 module, and the input data is divided into four feature sub-sections on average through the Se-Res2 module each time. The graphs are convolved separately, the convolved feature sub-maps are spliced, the spliced feature maps are again subjected to one-dimensional convolution to obtain the output of the hierarchical residual convolution, and the second feature is output by the last Se-Res2 module. picture;

In some embodiments of the present invention, the feature vector input by the feature aggregation module is the adjusted second feature map.

In the step of outputting the second feature map by the last Se-Res2 module, the second feature map output by the last Se-Res2 module is adjusted based on the preset compression excitation module, which specifically includes the steps:

The weight factor is obtained based on the preset compression excitation module, and the weight factor is weighted to the output feature map of each Se-Res2 module. The feature map weighted by the weight factor includes the second feature map output by the last Se-Res2 module. After adjustment The second feature map, the input of the previous Se-Res2 module is used as the input of the next Se-Res2 module, the output of each Se-Res2 module is weighted by the weight factor, and the second feature output by the last Se-Res2 module The graph is adjusted by weighting by weighting factors.

The feature extraction module introduces the mode of hierarchical residual connection in Res2Net, splits the features on the channel during one-dimensional hole convolution, extracts depression-related features at different scales, and then fuses the grouped features to improve the expressiveness of the network. Specifically, in each convolution module, after a one-dimensional convolution of the input feature map, the feature map is divided into four parts in order, represented by x _i , i∈{1,2,3,4} . Except for the feature sub-map x ₁ , each feature sub-map x _i is convolved by 3x3, and the result output after the convolution of the previous feature sub-map is added. For each feature sub-map x _i , the corresponding output y _i is as follows shown:

Among them, K _i represents the 3x3 convolution corresponding to the ith feature sub-map x _i , _yi represents the feature sub-map after convolution, i represents the sequence number of the feature sub-map, and y _i-1 represents the i-1 th feature sub-map Figure _xi-1 is the feature sub-map after convolution, and K _i represents the 3x3 convolution corresponding to the _i -th feature sub-map xi.

Each feature sub-map is merged after convolution, and the output of the hierarchical residual convolution is obtained after one-dimensional convolution.

Using the above scheme, a compressed excitation module is embedded in the one-dimensional hole convolution, which uses global information to evaluate the importance of each feature channel, that is, learns the weight information representing the importance of each channel, and readjusts the output of each channel after the convolution. characteristics, to achieve highlighting the more critical information for the diagnosis of depression, and suppress irrelevant redundant information. The compression excitation module is divided into two parts: compression, namely global information embedding and excitation, namely adaptive retuning. The global information embedding is to average the eigenvalues in the time domain to obtain the channel descriptor z, as shown below:

In adaptive retuning, a fully connected layer with a sigmoid activation function is used to obtain a normalized weight factor s to represent the importance of each channel.

s=σ ₁ (W ₂ f ₁ (W ₁ z+b ₁ )+b ₂ )

Among them, W ₁ , W ₂ , b ₁ , and b ₂ are the parameters of the two fully connected layers, respectively, f is the relu activation function, and σ is the sigmod activation function.

Finally, the weight factor is weighted to the features of each channel to complete the re-calibration of the original features in the channel dimension.

In some embodiments of the present invention, the feature aggregation module maps the convolved frame-level features into fixed-length segment-level features by calculating the mean and variance of the frame-level features.

With the above solution, the feature aggregation module will map the frame-level feature representation of the feature extraction module to the feature representation of the entire speech by stacking the mean and variance of each channel of the frame-level feature. Here the attention mechanism is introduced in the computer. Some speech frames contain more depressive cues and have a greater impact on the final result. By using the attention mechanism, these important frames can be given higher weights.

In some embodiments of the present invention, the classification module includes two fully connected layers and a Softmax layer, and outputs the probability value of the speech belonging to depression or normal people.

By adopting the above scheme, compared with the method of using the depression scale for diagnosis, the present invention does not need to rely on the experience of professional doctors for diagnosis, and does not require expensive infrastructure and complicated operation procedures. The present invention is based on the Mel frequency cepstral coefficient ( MFCC) feature extraction of speech features, using deep learning methods to process, segmenting long segments of speech, as time-delay neural network input classification results, and integrating to obtain depression diagnosis results.

In some embodiments of the present invention, before the step of inputting the Mel-frequency cepstral coefficients into the preset time-delay neural network model, the step further includes,

The feature data enhancement is performed on the Mel-frequency cepstral coefficients through a spectral mask, and the enhanced Mel-frequency cepstral coefficients are input into a preset time-delay neural network model.

In some embodiments of the present invention, the manner of spectral masking includes, but is not limited to, time-domain masking or frequency-domain masking.

(1) Time-domain masking: replace the adjacent frames in the Mel-frequency cepstral coefficient spectrogram with 0;

(2) Frequency-domain mask: Similar to the time-domain mask, several adjacent frequency bands are replaced with 0 in the frequency domain.

Using the above scheme, the spectral mask is used for feature data enhancement, and the data enhancement can expand the data sample size and improve the performance of the deep learning model.

In some embodiments of the present invention, in the step of separately calculating the short-term energy and short-term zero-crossing rate of each speech segment in the initial speech signal, the short-term energy is calculated based on the following formula:

Ex represents the short-term energy of speech segment _x , N represents the total number of frames in speech segment x, n represents any one of N frames, and x[n] represents the amplitude of the nth frame in N frames.

In some embodiments of the present invention, in the step of separately calculating the short-term energy and short-term zero-crossing rate of each speech segment in the initial speech signal, the short-term zero-crossing rate is calculated based on the following formula:

Z _x represents the short-term zero-crossing rate of the speech segment x, N represents the total number of frames in the speech segment x, n represents any one of the N frames, x(n) represents the amplitude of the nth frame in the N frames, x(n-1) represents the amplitude of the n-1th frame in N frames, and sgn represents the sign function;

In some embodiments of the present invention, when the voiced segment in the speech segment of the initial speech signal is acquired based on the short-term energy, the unvoiced segment in the speech segment of the initial speech signal is acquired based on the short-term zero-crossing rate Fragment steps,

Preset short-term energy threshold and short-term zero-crossing rate threshold;

Based on comparing the short-term energy value and the short-term energy threshold of each speech segment, the voiced segments in the speech segment are obtained;

Based on comparing the short-term zero-crossing rate value and the short-term zero-crossing rate threshold of each speech segment, the unvoiced segments in the speech segment are obtained.

In some embodiments of the present invention, the solution may be a way of directly comparing the short-term energy value and the short-term energy threshold, and the short-term zero-crossing rate value and the short-term zero-crossing rate threshold, so that the two can be compared to obtain speech segments voiced or unvoiced segments in ;

You can also use the following methods:

Set a short-term energy high threshold value T1 and low threshold value T2, and perform the first initial judgment. First, according to the starting point and end point set by the high threshold T1, and then according to T2 for the starting point and ending point of the selected range to the left. Right search, expand the range of voice selection, and can effectively detect continuous voiced segments by setting two thresholds;

According to the short-term zero-crossing rate of noise, a threshold T3 is set, and the range selected in the previous step is expanded forward and backward again, and the repeated regions are merged. All voiced segments and unvoiced segments in the initial speech signal are used to obtain valid speech segments.

In some embodiments of the present invention, pre-emphasis processing is performed on each of the valid speech segments based on the following formula;

y(n)=x(n)-αx(n-1)

x(n) represents the amplitude of the nth frame in the N frames, x(n-1) represents the amplitude of the n-1th frame in the N frames, and y(n) is the amplitude of the valid speech segment after pre-emphasis processing. The amplitude of the nth frame in the N frames, α is a pre-emphasis factor, in some embodiments of the present invention, α=0.97.

In some embodiments of the present invention, in the step of dividing the pre-emphasized effective speech segment into frames based on time to obtain multiple frame segments,

Each valid speech segment of the first time length is divided into a frame segment, and adjacent frame segments have overlapping segments of the second time length.

In some embodiments of the present invention, in the step of dividing the effective speech segment after pre-emphasis processing into frames based on time to obtain multiple frame segments, the signal is divided into frame segments of 25ms, in order to avoid frame segments between two frames. If the gap is too large, the boundary information is lost, and there is a 10ms overlap segment between the other two frames.

By adopting the above scheme, the boundary information is effectively preserved.

In some embodiments of the present invention, the step of calculating the Mel frequency cepstral coefficient corresponding to each frame segment includes:

Windowing is performed on each frame segment based on the window function;

Perform fast Fourier transform on the windowed frame segment to convert the time domain signal into a frequency domain signal;

Convert the frequency of the frequency domain signal to the Mel frequency based on the Mel filter to obtain the Mel frequency signal;

Perform inverse Fourier transform on the Mel-frequency signal, convert the Mel-frequency signal to the time domain, and obtain the Mel-frequency cepstral coefficient.

Using the above scheme, after dividing the signal into frames, each frame needs to be substituted into the window function, and the value outside the window is set to 0, so as to eliminate the discontinuity of the signal caused by the two ends of each frame;

Fast Fourier Transform, which converts time domain signals into frequency domain for subsequent frequency domain analysis. Since it is difficult to see the characteristics of the signal from the transformation of the signal in the time domain, the fast Fourier transform is carried out to convert it into the energy distribution in the frequency domain for analysis, and different energy distributions can represent different speech features;

The discrete cosine transform, where the inverse Fourier transform is performed, converts the generated Mel frequency domain signal to the time domain, and obtains the Mel frequency cepstral coefficients.

In some embodiments of the present invention, in the step of windowing processing each frame segment based on a window function, a Hamming window function is used;

The Hamming window function value is obtained according to the following formula:

w(a)=(1-α)-βcos[2πa/(A-1)];

w(a) represents the value of the Hamming window function, A represents the length of the window, a is the value anywhere in the window, and the β window parameter.

In some embodiments of the present invention, fast Fourier transform is performed on the windowed frame segment according to the following formula:

δ( _a ) represents the amplitude at a in the window length, and δa(k) represents the parameter value after fast Fourier transform.

In some embodiments of the present invention, the frequency of the frequency domain signal is converted to the Mel frequency based on the mel filter according to the following formula:

H _m (k) represents the frequency response of the Mel filter, M is the number of filters, 0≤m≤M; take the maximum frequency of 8kHz and the minimum frequency of 300Hz, and convert them to the Mel scale, which are 401.25Mel and 2834.99 respectively. Mel, select M points from the middle distance between the maximum frequency and the minimum frequency, and define them as f(1), f(2), ..., f(M) respectively, then f(0)=401.25, f(M+1 )=2834.99, f(0)<k<f(M+1), s(m) represents the logarithmic energy of the filter bank output, C(m) represents the Mel frequency, and g represents the Mel cepstral coefficient.

Mel filter group filtering, since the human ear has different sensitivity to different frequency signals, it usually pays more attention to the low frequency signal, so by using the Mel filter group to convert the original frequency signal to the Mel frequency, the triangular The filter is shown in Figure 5.

As shown in FIG. 4 , in some embodiments of the present invention, the feature extraction module includes a plurality of consecutive Se-Res2 modules, and each Se-Res2 module is provided with a Res2Net layer for convolution processing; the feature aggregation module It includes an attention mechanism layer, and calculates the mean and variance corresponding to each feature vector based on the attention mechanism; the classification module includes a sequence-connected fully connected layer and a Softmax layer, and the Softmax layer outputs probability parameters.

During the processing, the feature extraction module first averages the feature values in the time domain to generate the channel descriptor z, then calculates the weight of each channel, and finally multiplies the weight value s by the original feature to obtain the weighted feature.

In some embodiments of the present invention, in the step of calculating the mean and variance corresponding to each feature vector based on the attention mechanism:

According to the following formula, the scaling factor corresponding to each eigenvector is calculated and normalized:

e _t =v ^T f ₂ (Wh _t +b)+k;

e _t represents the attention score of the t-th frame segment, f ₂ represents the nonlinear activation function, W represents the weight parameter, h _t represents the feature vector of the t-th frame segment, b represents the bias parameter, v ^T and k are both The preset linear layer learning parameters, α _t represents the attention score normalized by softmax, and T represents the total number of frame segments;

Calculate the mean and variance corresponding to each eigenvector based on the scaling factor according to the following formula:

μ represents the mean, σ ₂ represents the variance, and t represents the t-th frame segment.

Using the above scheme, the feature aggregation module maps the convolved frame-level features into fixed-length segment-level features by calculating the mean and variance of the frame-level features. Here, an attention mechanism is introduced during the calculation. Certain speech frames contain more depressive cues and have a greater impact on the final result, and these important frames can be given higher weights by using the attention mechanism.

The time-delay neural network model of the present application uses the cross-entropy loss function to calculate the error between the output value of the network and the real value, uses the back-propagation algorithm to propagate the error value, and continuously optimizes and updates the weight of the network parameters.

In some embodiments of the present invention, the time-delay neural network model of the present application may be an ECAPA-TDNN network model.

In some embodiments of the present invention, due to the limitation of the input length of ECAPA-TDNN, after dividing each person's speech into multiple initial speech signals, each initial speech signal is used as the input of the model, and each initial speech signal is output. The probability parameter of the result of the signal, and finally combined with multiple results to output the result of whether the person suffers from depression, and finally each person will produce many segments of speech. Get a result of whether the person is depressed or not, integrating the predictions of the multi-speech speech produced by each person.

In some embodiments of the present invention, in the step of outputting the result of whether the person suffers from depression in combination with multiple results, it is also possible to assign a greater weight to the initial speech signal with a longer time, and calculate the result of the multiple results. The final prediction parameter is obtained by means of the weighted average of probability parameters, and the final prediction parameter is compared with the preset prediction threshold to obtain the result of whether the person suffers from depression;

Specifically, if the prediction parameter is greater than the preset prediction threshold, the person suffers from depression;

If the prediction parameter is not greater than the preset prediction threshold, the person does not suffer from depression.

To sum up, the voice data of depression is easy to obtain. It is only necessary to record the interview process between the patient and the doctor according to the diagnosis process, which is a convenient and fast way. The average prediction accuracy of the experiment of the present invention is 90.3%. The experimental results of the five experimental models have a small variation range, and the stability and accuracy are good in predicting depression, which proves the effectiveness of the method. The invention adopts artificial intelligence and speech signal processing technology to solve practical medical problems and has high practical value.

A second aspect of the present invention provides a device for automatic detection of speech depression based on a time-delay neural network, the device includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the The processor is configured to execute computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus implements the steps of the above method.

A third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the aforementioned automatic detection method for speech depression based on a time-delay neural network. The computer-readable storage medium may be a tangible storage medium such as random access memory (RAM), memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

It should be understood by those of ordinary skill in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is implemented in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, elements of the invention are programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted over a transmission medium or communication link by a data signal carried in a carrier wave.

It is to be understood that the present invention is not limited to the specific arrangements and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above-described embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the sequence of steps after comprehending the spirit of the present invention .

In the present invention, features described and/or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, and/or in combination with or in place of features of other embodiments Features of the implementation.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, various modifications and changes may be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A speech depression automatic detection method based on a time delay neural network is characterized by comprising the following steps,

acquiring an initial voice signal, dividing the initial voice signal into a plurality of voice sections, wherein each voice section comprises at least one voice frame, and respectively calculating the short-time energy and the short-time zero crossing rate of each voice section in the initial voice signal;

obtaining a voiced segment in a voice segment of the initial voice signal based on the short-time energy, obtaining an unvoiced segment in the voice segment of the initial voice signal based on the short-time zero crossing rate, and combining all voiced segments and unvoiced segments in the initial voice signal to obtain an effective voice segment;

carrying out pre-emphasis processing on each effective voice fragment, framing the pre-emphasized effective voice fragments based on time to obtain a plurality of frame fragments, and calculating a Mel frequency cepstrum coefficient corresponding to each frame fragment;

inputting the Mel frequency cepstrum coefficient into a preset time delay neural network model, calculating to obtain a feature vector corresponding to the Mel frequency cepstrum coefficient based on a feature extraction module of the time delay neural network model, calculating to obtain a mean value and a variance corresponding to each feature vector based on a feature aggregation module of the time delay neural network model, and inputting the mean value and the variance corresponding to each feature vector into a classification module of the time delay neural network model to obtain a probability parameter.

2. The method for automatically detecting speech depression based on time delay neural network as claimed in claim 1, further comprising a step before the step of inputting the Mel frequency cepstral coefficient into a preset time delay neural network model,

and enhancing the characteristic data of the Mel frequency cepstrum coefficient through a frequency spectrum mask, and inputting the enhanced Mel frequency cepstrum coefficient into a preset time delay neural network model.

3. The method according to claim 1, wherein in the step of separately calculating the short-term energy and the short-term zero-crossing rate of each speech segment in the initial speech signal, the short-term energy is calculated based on the following formula:

E _x representing the short-time energy of a speech segment x, N representing the total number of frames in the speech segment x, N representing any of the N frames, x [ N ]]Representing the amplitude of the nth frame of the N frames;

in the step of calculating the short-time energy and the short-time zero-crossing rate of each speech segment in the initial speech signal respectively, the short-time zero-crossing rate is calculated based on the following formula:

Z _x represents the short-time zero-crossing rate of a speech segment x, N represents the total number of frames in the speech segment x, N represents any one of N frames, x (N) represents the amplitude of the nth frame in the N frames, x (N-1) represents the amplitude of the nth-1 frame in the N frames, and sgn represents a sign function.

4. The method according to claim 1, wherein in the steps of obtaining voiced segments of the speech segments of the initial speech signal based on the short-time energy, obtaining unvoiced segments of the speech segments of the initial speech signal based on the short-time zero-crossing rate,

presetting a short-time energy threshold and a short-time zero-crossing rate threshold;

obtaining voiced sound segments in the voice segments based on comparing the short-time energy value and the short-time energy threshold value of each voice segment;

and acquiring unvoiced segments in the voice segments based on the comparison of the short-time zero-crossing rate value and the short-time zero-crossing rate threshold of each voice segment.

5. The method as claimed in claim 1, wherein the step of framing the pre-emphasized effective speech segment based on time to obtain a plurality of frame segments, and calculating mel-frequency cepstrum coefficients corresponding to each frame segment comprises:

in the step of framing the pre-emphasized effective speech segment based on time to obtain a plurality of frame segments,

dividing each effective voice fragment with a first time length into a frame fragment, wherein adjacent frame fragments have a superposition section with a second time length;

windowing each frame segment based on a window function;

performing fast Fourier transform on the windowed frame segment, and converting a time domain signal into a frequency domain signal;

converting the frequency of the frequency domain signal to a Mel frequency based on a Mel filter to obtain a Mel frequency signal;

and performing inverse Fourier transform on the Mel frequency signal, and converting the Mel frequency signal into a time domain to obtain a Mel frequency cepstrum coefficient.

6. The automatic voice depression detection method based on the time delay neural network as claimed in any one of claims 1 to 5, wherein the feature extraction module comprises a plurality of continuous Se-Res2 modules, each Se-Res2 module is provided with a Res2Net layer for convolution processing, and the feature extraction module adopts layered residual connection to extract frame-level features; the feature aggregation module comprises an attention mechanism layer, and the mean value and the variance corresponding to each feature vector are calculated based on the attention mechanism; the classification module comprises a full connection layer and a Softmax layer which are connected in sequence, and probability parameters are output by the Softmax layer.

7. The method for automatically detecting speech depression based on time delay neural network as claimed in claim 6, wherein in the step of extracting frame-level features by the feature extraction module using hierarchical residual connection:

the method comprises the following steps of changing a Mel frequency cepstrum coefficient into a first feature graph through one-dimensional convolution size adjustment, inputting the first feature graph into a Se-Res2 module, averagely dividing input data into four feature sub-graphs through a Se-Res2 module each time, respectively convolving the feature sub-graphs, splicing the convolved feature sub-graphs, obtaining output of hierarchical residual convolution through one-dimensional convolution of the spliced feature graph again, outputting a second feature graph through a last Se-Res2 module, and convolving the feature sub-graphs according to the following formula:

y _i representing the convolved feature subgraphs, i representing the serial number of the feature subgraphs, y _i-1 Represents the i-1 th characteristic sub-graph x _i-1 Convolved feature subgraph, K _i Representing the ith characteristic sub-graph x _i The corresponding 3x3 convolution.

8. The automatic voice depression detection method based on the time delay neural network as claimed in claim 7, wherein in the step of outputting the second feature map by the last Se-Res2 module, the step of adjusting the output second feature map by the last Se-Res2 module based on the preset compressed excitation module comprises the steps of,

weighting the weighting factors to the output characteristic diagram of each Se-Res2 module based on the weighting factors obtained by the preset compressed excitation module, wherein the characteristic diagram weighted by the weighting factors comprises a second characteristic diagram output by the last Se-Res2 module, and the adjusted second characteristic diagram is obtained:

obtaining a weight factor based on a preset compression excitation module according to the following formula:

s＝σ ₁ (W ₂ f ₁ (W ₁ z+b ₁ )+b ₂ )

z is a channel descriptor, R represents the total number of frames of the first feature map, R represents the R-th frame in the R frames, and gamma _r Feature vector, W, representing the mth frame of the first feature map ₁ 、W ₂ 、b ₁ 、b ₂ Respectively the parameters of two fully-connected layers, f ₁ For relu activation function, σ ₁ For sigmod activation functions, s denotes the weighting factor.

9. The method for automatically detecting the speech depression based on the time-delay neural network as claimed in claim 1, wherein in the step of calculating the mean and the variance corresponding to each feature vector based on the attention mechanism:

calculating a scaling factor corresponding to each feature vector according to the following formula, and normalizing;

e _t ＝v ^T f ₂ (Wh _t +b)+k；

e _t denotes the attention score, f, of the t-th frame segment ₂ Denotes a non-linear activation function, W denotes a weight parameter, h _t Feature vector representing the t-th frame segment, b bias parameter, v ^T And k are the preset parameters of linear layer learning, alpha _t Expressed as attention score normalized by softmax, T represents the total number of frame segments;

calculating the mean value and the variance corresponding to each feature vector based on the scaling factors according to the following formula;

μ denotes mean, σ ₂ Representing the variance, and t represents the t-th frame segment.

10. An apparatus for automatic voice depression detection based on a time-delay neural network, the apparatus comprising a computer device, the computer device comprising a processor and a memory, the memory having stored therein computer instructions, the processor being configured to execute the computer instructions stored in the memory, the apparatus implementing the steps of the method according to any one of claims 1-9 when the computer instructions are executed by the processor.

Images