Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Apr 3;14(7):1819.
doi: 10.3390/cancers14071819.

System for the Recognizing of Pigmented Skin Lesions with Fusion and Analysis of Heterogeneous Data Based on a Multimodal Neural Network

Pavel Alekseevich Lyakhov  1   2 , Ulyana Alekseevna Lyakhova  3 , Nikolay Nikolaevich Nagornov  2
Affiliations

System for the Recognizing of Pigmented Skin Lesions with Fusion and Analysis of Heterogeneous Data Based on a Multimodal Neural Network

Pavel Alekseevich Lyakhov et al. Cancers (Basel). .

Abstract

Today, skin cancer is one of the most common malignant neoplasms in the human body. Diagnosis of pigmented lesions is challenging even for experienced dermatologists due to the wide range of morphological manifestations. Artificial intelligence technologies are capable of equaling and even surpassing the capabilities of a dermatologist in terms of efficiency. The main problem of implementing intellectual analysis systems is low accuracy. One of the possible ways to increase this indicator is using stages of preliminary processing of visual data and the use of heterogeneous data. The article proposes a multimodal neural network system for identifying pigmented skin lesions with a preliminary identification, and removing hair from dermatoscopic images. The novelty of the proposed system lies in the joint use of the stage of preliminary cleaning of hair structures and a multimodal neural network system for the analysis of heterogeneous data. The accuracy of pigmented skin lesions recognition in 10 diagnostically significant categories in the proposed system was 83.6%. The use of the proposed system by dermatologists as an auxiliary diagnostic method will minimize the impact of the human factor, assist in making medical decisions, and expand the possibilities of early detection of skin cancer.

Keywords: convolutional neural networks; dermatoscopic images; digital image processing; hair removal; heterogeneous data; melanoma; metadata; multimodal neural networks; pattern recognition; pigmented skin lesions.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure A1
Figure A1
Graph of learning outcomes of a multimodal neural network system for classifying dermatoscopic images of pigmented skin lesions based on CNN AlexNet: (a) loss function; (b) recognition accuracy.
Figure A2
Figure A2
Graph of learning outcomes of a multimodal neural network system for classifying dermatoscopic images of pigmented skin lesions based on CNN SqueezeNet: (a) loss function; (b) recognition accuracy.
Figure A3
Figure A3
Graph of learning outcomes of a multimodal neural network system for classifying dermatoscopic images of pigmented skin lesions based on CNN ResNet-101: (a) loss function; (b) recognition accuracy.
Figure 1
Figure 1
Multimodal neural network system for the classification of dermatoscopic images of pigmented skin lesions with preliminary heterogeneous data processing.
Figure 2
Figure 2
Examples of pigmented skin lesions images with hairy structures: (a) vascular lesions; (b) nevus; (c) solar lentigo; (d) dermatofibroma; (e) seborrheic keratosis; (f) benign keratosis; (g) actinic keratosis; (h) basal cell carcinoma; (i) squamous cell carcinoma; (j) melanoma.
Figure 3
Figure 3
Scheme of the proposed method of identification and hair removal from dermatoscopic images of pigmented skin lesions.
Figure 4
Figure 4
Images obtained as a result of passing each stage of the method of identification and hair removal: (a) input RGB image PRGB; (b) the color component PR, presented in shades of gray; (c) the result of the HR3 closing operation; (d) the result of the subtraction operation HR2 (inverted image); (e) the result of zeroing pixels HR1 (inverted image); (f) the result of the HR dilatation operation (inverted image); (g) pixel interpolation result PR*; (h) output RGB image PRGB*. Scale bar or magnification.
Figure 5
Figure 5
Metadata pre-processing scheme.
Figure 6
Figure 6
Neural network architecture for multimodal classification of pigmented skin lesions based on CNN AlexNet. Scale bar or magnification.
Figure 7
Figure 7
Diagram of the distribution of the number of dermatoscopic images in 10 diagnostically significant categories.
Figure 8
Figure 8
Diagrams of the distribution of the base of dermatoscopic images according to the statistical data of patients: (a) by gender; (b) by age; (c) by the location of the pigmented lesion on the body.
Figure 9
Figure 9
Examples of identification and cleaning of hair structures from dermatoscopic images of pigmented skin lesions using the proposed method: (a) original dermatoscopic image; (b) the result of extracting hair in the image (inverted image); (c) dermatoscopic image cleared of hair structures. Scale bar or magnification.
Figure 10
Figure 10
An example of pre-processing statistical patient metadata using one-hot encoding.
Figure 11
Figure 11
Images obtained as a result of affine transformations: (a) original image; (b) image after the operation of rotation by a given angle; (c) image after shift operation; (d) image after the scaling operation; (e) image after the reflection operation. Scale bar or magnification.
Figure 12
Figure 12
Examples of dermatoscopic training images that have been previously cleaned and enlarged using affinity transformations. Scale bar or magnification.
Figure 13
Figure 13
Confusion matrix in the testing results in a multimodal neural network system for recognizing pigmented skin lesions based on CNN AlexNet.
Figure 14
Figure 14
Confusion matrix in the testing results in a multimodal neural network system for recognizing pigmented skin lesions based on CNN SqueezeNet.
Figure 15
Figure 15
Confusion matrix in the testing results in a multimodal neural network system for recognizing pigmented skin lesions based on CNN ResNet-101.
Figure 16
Figure 16
The confusion matrix of the test results of the proposed multimodal neural network system based on CNN is divided into two groups: (a) AlexNet; (b) SqueezeNet; (c) ResNet-101.
Figure 17
Figure 17
Classification table neural network systems for recognizing pigmented skin lesions for analysis McNemar based on CNN: (a) AlexNet; (b) SqueezeNet; (c) ResNet-101.
Figure 18
Figure 18
Receiver operative characteristics (ROC) curve when testing a multimodal neural network system for recognizing pigmented lesions and skin based on CNN: (a) AlexNet; (b) SqueezeNet; (c) ResNet-101.
Figure 18
Figure 18
Receiver operative characteristics (ROC) curve when testing a multimodal neural network system for recognizing pigmented lesions and skin based on CNN: (a) AlexNet; (b) SqueezeNet; (c) ResNet-101.
See this image and copyright information in PMC

References

    1. Health Consequences of Excessive Solar UV Radiation. [(accessed on 18 October 2021)]. Available online: https://www.who.int/news/item/25-07-2006-health-consequences-of-excessiv....
    1. Rogers H.W., Weinstock M.A., Harris A.R., Hinckley M.R., Feldman S.R., Fleischer A.B., Coldiron B.M. Incidence Estimate of Nonmelanoma Skin Cancer in the United States, 2006. Arch. Dermatol. 2010;146:283–287. doi: 10.1001/archdermatol.2010.19. - DOI - PubMed
    1. Madan V., Lear J.T., Szeimies R.-M. Non-Melanoma Skin Cancer. Lancet. 2010;375:673–685. doi: 10.1016/S0140-6736(09)61196-X. - DOI - PubMed
    1. The Skin Cancer Foundation Skin Cancer Facts & Statistics. [(accessed on 21 October 2021)]. Available online: https://www.skincancer.org/skin-cancer-information/skin-cancer-facts/
    1. Stern R.S. Prevalence of a History of Skin Cancer in 2007: Results of an Incidence-Based Model. Arch. Dermatol. 2010;146:279–282. doi: 10.1001/archdermatol.2010.4. - DOI - PubMed

LinkOut - more resources