Using AI and Polarized Imaging to Assist Physicians with Early Skin Cancer Diagnosis

Andrew Pun

Age 18 | Scarborough, ON

First Place winner for 2021 Weston Youth Innovation Award | Ted Rogers Innovation Award | Youth Science Canada Online STEM Fair 2020 YSC Disease and Illness Challenge Ribbon, National Ribbon and Regional Ribbon

Edited by: Veronica Xia

Skin cancer is the most common type of cancer worldwide. In order to increase the chances of survival, doctors must accurately diagnose skin cancers such as melanoma early so that they do not spread to other parts of the body. There are a few innovations that can help this process. During examinations, doctors sometimes use a special type of light called polarized light to illuminate skin lesions, which reduces glare and reveals deeper structures of the lesion. Additionally, there are artificial intelligence models developed to analyze a picture of a lesion and diagnose it with a condition. I wanted to combine these ideas to investigate if an AI would more accurately diagnose skin lesion images if they were captured using polarized light. I developed an artificial intelligence model to recognize several types of skin lesions from images. Then I created a 3D-printed device that allows smartphone cameras to take a picture of skin lesions using polarized light, and used it to obtain pictures of skin lesions both with and without polarization. I then gave these images to the AI to determine if it was better at recognizing polarized images or non-polarized images. I observed that in three out of the four categories of skin lesions, the AI performed better when classifying polarized images than non-polarized images, while the accuracy was unaffected for the fourth category. However, I found that both polarized and non-polarized images have their own benefits, so using both would maximize a physician’s ability to formulate a more accurate diagnosis. I believe my biomedical device has the potential to help doctors better visualize their patients’ lesions, while the AI I wrote could provide them with an additional opinion for a diagnosis.

INTRODUCtion

Skin cancer is the most common type of cancer worldwide, and more than two people die from skin cancer every hour (Rogers, et al. 2015). Melanoma is the deadliest type of skin cancer and is the leading cause of 83% of skin cancer-related deaths (Tannous, Al-Arashi, Shah, and Yaroslavsky, 2009). In order to increase the chances of successful treatment, it is paramount that efforts are made by a physician to accurately diagnose melanoma and other skin-related cancers in a patient before they metastasize (Apalla, et al. 2017).

Artificial intelligence models are currently under development to detect melanoma and other skin cancers in their early stages, and the most cutting-edge algorithms can classify validated datasets at ~95 percent accuracy (European Society for Medical Oncology, 2018). However, this is only the case if the data is captured under optimal circumstances for maximal clarity (American Academy of Dermatology, 2019). Since dermatologist-level classifiers are trained using clinical datasets to recognize skin lesions from dermoscopies (Esteva et al. 2017), one would need to take a picture of their own skin lesion using dermoscopic imaging techniques to obtain the best result from the classifier; it would be difficult for classifiers to correctly identify unclear images of skin lesions, regardless of how accurate the model is in practice (Freeman et al. 2020).

One such imaging technique, called polarized dermoscopy, reduces glare and makes some skin structures, such as white spots that appear in basal cell carcinoma (Rosendahl and Marozava, 2019), and white lines that hint at the presence of melanoma or dermatofibroma (Cohen, et al. 2014), more visible. The light emitted from the source is first polarized linearly by a filter (Louie et al. 2018). When this light contacts the skin, a portion of it is reflected by the stratum corneum (the outermost layer of the epidermis), but the remaining light illuminates deeper layers of the dermis. A portion of the light is then backscattered in a perpendicular orientation, which is then allowed to pass through a polarizing filter in front of the optic, which blocks out the light reflected by the stratum corneum (Pan et al. 2008).

The first purpose of the project is to create a system that would assist physicians in accurately diagnosing skin lesions. The system consisted of an artificial intelligence model that classifies skin lesions, a custom polarizing light biomedical device, and a web application interface. The artificial intelligence model was trained using the publicly available Harvard HAM10000 clinical dermoscopic dataset. Included is an image set known as the Australian Rosendahl image set, which uses pictures of skin lesions, many of which have been obtained by polarized dermoscopy methods (Tschandl, et al. 2018). After the model was created, it was integrated into the web application interface, allowing it to be accessible from a smartphone web browser. A custom polarizing light biomedical device (referred to as the Polarizer Device) was created and designed to be attached to a smartphone. Its function was to grant the smartphone’s camera polarized-imaging capabilities for capturing skin lesions, in order to emulate non-contact, polarized dermoscopy techniques. As a tool for physicians to use, the web interface allows a physician to upload its diagnoses of skin lesions into their electronic medical record of choice, such as the McMaster OSCAR Electronic Medical Record. These products in the system were designed to be used in tandem with each other: a physician would be able to use their smartphone with the Polarizer Device to capture polarized light images of a patient’s skin lesion, and through the web application be able to upload the image to the artificial intelligence model, which analyzes the lesion and provides its own diagnosis, so that it can be directly added into the patient’s file.

The second purpose is to investigate whether or not the accuracy of the artificial intelligence model was improved when classifying images captured by the smartphone with the Polarizer Device as opposed to regular, non-polarized images taken without the device’s filters. As the artificial intelligence model was trained using dermoscopy images, including polarized images, it was expected that it would classify pictures taken using polarized imaging techniques more accurately than non-polarized skin lesions images.

HYPOTHESIS

It is hypothesized that by using polarized imaging to take pictures of skin lesions, the performance of the artificial intelligence classifier will improve significantly. As the model has been trained to primarily recognize dermoscopic images (including polarized images) due to being trained using the HAM10000 dataset, it is expected that it would have more confidence in the correct diagnosis when classifying images that have been captured using the smartphone with the Polarizer Device.

MATERIALS

An Ubuntu computer was used for the software side of the project. Python3 was the main language used for writing the AI. Bash, PHP and HTML were used to make the web interface. The HAM10000 dataset by Harvard was used for training the AI. The Polarizer Device was mainly made of 3D-printed parts, but also included linearly polarizing filters, a DC-DC 3V voltage regulator, 4 AA batteries, a 2-position electrical switch, and a white LED light. A soldering iron was used to assemble the circuitry and hot glue was used to glue together the 3D-printed parts. The Polarizer Device was attached to a smartphone, to allow its camera to take polarized images.

METHODS

Part 1: Creating the Polarizer Device
I created a 3-D printed polarizing light biomedical device intended for a smartphone (Google Pixel 3A).  The device, which is magnetically attached to a smartphone, is composed of an LED light source and has two intermediary linearly polarizing filters, positioned perpendicular to each other. One of the filters fits in front of the smartphone’s camera, while the other fits in front of the light source. The light source was created using a white LED light from a night light, wired in series to an on and off switch, four AA batteries, and a voltage regulator that kept the voltage at 2.96V DC. The light source and camera filter are adjustable via a sliding dovetail rail to be compatible with different smartphones. The filters are also easily removable.

Figure 1. The Polarizer Device attached to a smartphone.

Figure 2. A diagram displaying the features and parts of the Polarizer Device.

Part 2: Artificial Intelligence Model and Interface
I developed a machine learning classifier using the artificial intelligence open-source software library, Google TensorFlow, and trained with data from the aforementioned Harvard HAM10000 skin lesion dataset. The artificial intelligence model uses the Adam Optimization Algorithm (Kingma and Ba, 2014). A Python program was written to read the model and was implemented into a web application. The web application has been written using an HTML frontend, and an image upload program was created using PHP. This web-based infrastructure allows a user to easily upload pictures from a web browser, such as a browser on a smartphone. The web application is able to securely pass a picture to the artificial intelligence model for analysis and display the results of the model’s diagnosis.

Part 3: Data Collection
A qualified physician used a Google Pixel 3A smartphone with the Polarizer Device to capture images of consenting participants’ skin lesions at his clinic, both with and without light polarization. The images were taken in a dark room; the light from the Polarizer Device was the only source of light. All pictures taken of skin lesions were anonymized; they had no identifiable information attached to them such as names, phone numbers or patient chart numbers, and no identifiable features such as faces or tattoos, etc. Each picture was rotated in 4 orientations and fed into the model for analysis, and diagnoses for the skin lesions were provided by a dermatologist, pathologist, and family physician.

Figure 3. A comparison between a skin lesion taken with and without the Polarizer Device. Note that with the polarization, the lesion has reduced glare and its subsurface structures are more visible.

Figure 4. The Polarizer Device being used to take a picture of a skin lesion.

Part 4: Web Application Interface Test and Device Effectiveness
The final part of the procedure was meant for testing the web application interface and evaluating the effectiveness of these biomedical devices. The artificial intelligence model was configured to upload the results of the data analysis to a demo OSCAR Electronic Medical Record system. To observe if this configuration is functional, fake patient files were created, with false patient chart numbers and other information. To protect the privacy of participants, no images from participants were used to test the web application interface. Miscellaneous images of skin conditions found on the public domain were used and uploaded to the patient’s file. A Bash script was written to detect and upload these images from the patient’s file to the artificial intelligence model, which returned a text file of the model’s analysis of the image.

results

Figure 5. The table of the results with the four different lesion types. It was found that in dermatofibroma, melanocytic nevi, and vascular lesions, there were more instances with higher polarized confidence than non-polarized confidence, although the amount is most significant with vascular skin lesions. Benign keratosis had an equal amount of lesions that had higher polarized confidence as lesions that had higher non-polarized confidence. x̄ indicates the mean confidence values of the lesions, and σx indicates the standard deviation of the data of the lesions, and SNR indicates the signal to noise ratio, calculated by x̄/σx. Typically, a lower SNR indicates that the deviation is high compared to the mean.

Figure 6. The confidence levels for polarized and non-polarized images for individual lesions were mapped in four area graphs, each graph for the four types of lesions. Blue bars represent the confidence of non-polarized images, while red bars indicate the confidence of polarized images. Bars that line up vertically are the corresponding polarized/non-polarized confidences of the same individual lesion.

Of the 57 lesions, there were 12 benign keratoses, 5 dermatofibromas, 37 melanocytic nevi, and 3 vascular lesions. There were no instances of melanoma, basal cell carcinoma, or actinic keratosis as none of the physician’s patients were clinically diagnosed with those lesions. I organized these 4 lesion categories in a table and marked down the number of individual lesions that had a higher polarized confidence than non-polarized confidence for each category, in the form of a percentage. I also calculated the mean confidence and standard deviation for polarized and non-polarized images for each category and was able to calculate the signal-to-noise ratio (SNR). A set of graphs was made to map the difference in confidence between the polarized and non-polarized images for individual lesions in a category. A positive value signifies a higher confidence in polarized images, while a negative value signifies a higher confidence value in non-polarized images. I chose to do this to easily visualize each individual lesion to spot any patterns, rather than just merely drawing conclusions from mean values.

DISCUSSION

It was found that polarized images had higher confidence than non-polarized images in 50.0% of benign keratoses, 65.0% of dermatofibromas, 68.2% of melanocytic nevi, and 91.7% of vascular skin lesions. When comparing the mean values, benign keratoses and dermatofibromas had a higher confidence in non-polarized images, while melanocytic nevi and vascular skin lesions had a higher confidence in polarized images. However, in most cases, the signal-to-noise ratio was very low, meaning that the deviation was very high compared to the mean.

The reason that some categories of lesions benefitted from polarization more than others is probably due to their different physical appearances. Polarization reduces glare and reveals subsurface structures, but also blurs some lesion’s borders and diminishes their perceived elevation/texture, two important characteristics for diagnosing a skin lesion. What likely happened was that with benign keratoses, polarization reduced their characteristic bumpy texture and made it harder to visually distinguish from melanocytic nevi. This would explain why with benign keratoses, the effects of polarization were not as prominent as other types of lesions.

Polarized dermoscopy is a unique way of improving the clarity of a skin lesion, by reducing glare and revealing subsurface structures via polarized light. This project combined this imaging technique with the recent innovation of artificial intelligence skin-lesion classifiers. The rationale behind this was that many of these classifiers are trained off of medical datasets, and most of these datasets include images taken with polarized dermoscopy, such as the HAM10000 dataset (Tschandl, et al. 2018). So in theory, a classifier would be more familiar with recognizing images of skin lesions taken using the polarized dermoscopy technique. That is what this project sought to find out, as it has not been investigated before.

Apart from investigating whether or not polarized dermoscopic imaging would improve accuracy, this project developed a system that could assist physicians with accurately diagnosing skin lesions. The polarizer device could be used by physicians to take clearer pictures of their patients’ lesions, and the artificial intelligence model could quickly formulate its own diagnosis of a lesion, providing a second opinion for the physician.

However, it is important to remember that an artificial intelligence model is only a tool meant to assist a physician, and not to replace a pathologist or dermatologist. The diagnosis from a family physician, pathologist, or dermatologist should always supersede the artificial intelligence model’s classification of a lesion. Unlike this AI, physicians have access to the history of a patient, which is an important factor for formulating a diagnosis. If there is a discrepancy in the diagnoses between the artificial intelligence model and a professional, the physician should re-evaluate the lesion again.

FUTURE STEPS

In the future, I would extend my project by introducing circularly polarized light to the experiment and seeing how it affects accuracy compared to linearly polarized light. I would also improve my device by adding a magnification lens, which would allow me to take higher resolution images.

The project could also benefit from an AI that has been trained with more types of lesions, such as dysplastic nevi or cherry angiomas. Additionally, if the AI was trained with more images, its accuracy may improve. Furthermore, I would try to obtain a bigger sample size for the experiment portion, with more types of skin lesions, such as the inclusion of melanoma and basal cell carcinoma.

CONCLUSIONS

In conclusion, my hypothesis was partially supported, as three out of the four types of lesions were more accurately classified by the AI when they were polarized. While all polarized lesions featured had reduced glare and more visible subsurface structures, some may have had less defined borders and their texture wasn’t as prominent due to the lack of glare. For benign keratoses, in particular, this may have reduced the AI’s accuracy as one of the lesions' defined characteristics is its bumpy surfaces.

Polarized and non-polarized images seem to complement each other by bringing out different aspects of a lesion, so when it comes to human analysis, both may be necessary to formulate a more accurate diagnosis, as one image may reveal a feature of the lesion that the other image lacks.

Additional information and figures for the project can be found here: https://projectboard.world/ysc/project/using-ai-and-polarized-imaging-to-assist-physicians-with-early-skin-cancer-diagnosis

ACKNOWLEDGEMENTS

I would like to thank my high school science teacher Mr. Henri van Bemmel, who was my adult sponsor for one of the science fairs this project was submitted to. He provided valuable suggestions for my project and guided me through the fair’s application process.

REFERENCES

American Academy of Dermatology Association. (2019, March 1). Artificial intelligence shows promise for skin cancer detection [Review of Artificial intelligence shows promise for skin cancer detection]. American Academy of Dermatology Association; American Academy of Dermatology Association. https://www.aad.org/news/ai-and-skin-cancer-detection

Apalla, Z., Nashan, D., Weller, R. B., & Castellsagué, X. (2017). Skin Cancer: Epidemiology, Disease Burden, Pathophysiology, Diagnosis, and Therapeutic Approaches. Dermatology and Therapy, 7(S1), 5–19. https://doi.org/10.1007/s13555-016-0165-y

Braun, R. P., Scope, A., Dimitriou, F., Reiter, O., & Marghoob, A. A. (2019, April 8). Polarized dermoscopy (R. Braun, Ed.) [Review of Polarized dermoscopy]. Dermoscopedia; Dermoscopedia. https://dermoscopedia.org/w/index.php?title=Polarized_dermoscopy&oldid=15355

Canadian Cancer Society. (n.d.). Finding skin cancer early [Review of Finding skin cancer early]. Canadian Cancer Society; Canadian Cancer Society. Retrieved December 9, 2019, from https://www.cancer.ca/en/cancer-information/cancer-type/skin-melanoma/finding-cancer-early/?region=bc

Cohen, Y., Elpern, D., Wolpowitz, D., & Rosendahl, C. (2014). Glowing in the dark: case report of a clue-poor melanoma unmasked by polarized dermatoscopy. Dermatology Practical & Conceptual. https://doi.org/10.5826/dpc.0401a14

Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017). Erratum: Corrigendum: Dermatologist-level classification of skin cancer with deep neural networks. Nature, 546(7660), 686–686. https://doi.org/10.1038/nature22985

European Society for Medical Oncology. (2018, May 28). Man Against Machine: Artificial Intelligence is Better than Dermatologists at Diagnosing Skin Cancer [Annals of Oncology Press Release] [Review of Man Against Machine: Artificial Intelligence is Better than Dermatologists at Diagnosing Skin Cancer [Annals of Oncology Press Release]]. European Society for Medical Oncology; European Society for Medical Oncology. https://www.esmo.org/Press-Office/Press-Releases/Artificial-Intelligence-Skin-Cancer-Diagnosis

Freeman, K., Dinnes, J., Chuchu, N., Takwoingi, Y., Bayliss, S. E., Matin, R. N., Jain, A., Walter, F. M., Williams, H. C., & Deeks, J. J. (2020). Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. BMJ, m127. https://doi.org/10.1136/bmj.m127

Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. ArXiv.org. https://arxiv.org/abs/1412.6980

Louie, D. C., Phillips, J., Tchvialeva, L., Kalia, S., Lui, H., Wang, W., & Lee, T. K. (2018). Degree of optical polarization as a tool for detecting melanoma: proof of principle. Journal of Biomedical Optics, 23(12), 1. https://doi.org/10.1117/1.jbo.23.12.125004

Pan, Y., Gareau, D. S., Scope, A., Rajadhyaksha, M., Mullani, N. A., & Marghoob, A. A. (2008). Polarized and Nonpolarized Dermoscopy. Archives of Dermatology, 144(6). https://doi.org/10.1001/archderm.144.6.828

Rogers, H. W., Weinstock, M. A., Feldman, S. R., & Coldiron, B. M. (2015). Incidence Estimate of Nonmelanoma Skin Cancer (Keratinocyte Carcinomas) in the U.S. Population, 2012. JAMA Dermatology, 151(10), 1081–1086. https://doi.org/10.1001/jamadermatol.2015.1187

Rosendahl, C., & Marozava, A. (2019). Dermatoscopy and skin cancer: a handbook for hunters of skin cancer and melanoma. Scion Publishing Limited.

Tannous, Z., Al-Arashi, M., Shah, S., & Yaroslavsky, A. N. (2009). Delineating melanoma using multimodal polarized light imaging. Lasers in Surgery and Medicine, 41(1), 10–16. https://doi.org/10.1002/lsm.20736

Tschandl, P., Rosendahl, C., & Kittler, H. (2018). The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1). https://doi.org/10.1038/sdata.2018.161

BIBLIOGRAPHY

Cohen, Y., Elpern, D., Wolpowitz, D., & Rosendahl, C. (2014). Glowing in the dark: case report of a clue-poor melanoma unmasked by polarized dermatoscopy. Dermatology Practical & Conceptual. https://doi.org/10.5826/dpc.0401a14

Marghoob, A. A. (2019, May 22). Two-step algorithm (R. P. Braun, Ed.) [Review of Two-step algorithm]. Dermoscopedia; Dermoscopedia. https://dermoscopedia.org/w/index.php?title=Two-step_algorithm&oldid=15900

Singh, P., & Manure, A. (2019). Neural Networks and Deep Learning with TensorFlow. Learn TensorFlow 2.0, 53–74. https://doi.org/10.1007/978-1-4842-5558-2_3

Tschandl, P., Rosendahl, C., & Kittler, H. (2018). The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1). https://doi.org/10.1038/sdata.2018.161