Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks

Roman C. Maron; Michael Weichenthal; Jochen S. Utikal; A. Hekler; Carola Berking; Axel Hauschild; Alexander H. Enk; Sebastian Haferkamp; Joachim Klode; Dirk Schadendorf; Philipp Jansen; Tim Holland-Letz; Bastian Schilling; Christof von Kalle; Stefan Fröhling; Maria R. Gaiser; Daniela Hartmann; Anja Gesierich; Katharina C. Kähler; Ulrike Wehkamp; Ante Karoglan; Claudia Bär; Titus J. Brinker; Laurenz Schmitt; Wiebke K. Peitsch; Friederike Hoffmann; Jürgen C. Becker; Christina Drusio; G. Lodde; Stefanie Sammet; Wiebke Sondermann; S. Ugurel; Jeannine Zader; Alexander Enk; Martin Salzmann; S. Schäfer; Knut Schäkel; J. Winkler; Priscilla Wölbing; Hiba Asper; Ann Sophie Bohne; Victoria Brown; Bianca Burba; Sophia Deffaa; Cecilia Dietrich; Matthias Dietrich; Katharina Antonia Drerup; Friederike Egberts; Anna Sophie Erkens; S. Greven; V. Harde; Marion Jost; M. Kaeding; Katharina Kosova; Stephan Lischner; M. Maagk; Anna Laetitia Messinger; M. Metzner; Rogina Motamedi; Ann Christine Rosenthal; Ulrich Seidl; Jana Stemmermann; Kaspar Torz; Juliana Giraldo Velez; Jennifer Haiduk; Mareike Alter; Paul Bergenthal; Anne Gerlach; Christian Holtorf; Sophie Kindermann; L. Kraas; Moritz Felcht; Claus Detlev Klemke; Hjalmar Kurzen; Thomas Leibing; Verena Müller; Raphael R. Reinhard; Jochen Utikal; Franziska Winter; Laurie Eicher; Markus Heppt; Katharina Kilian; Sebastian Krammer; D. Lill; Anne Charlotte Niesert; Eva Oppel; Elke Sattler; S. Senner; Jens Wallmichrath; Hans Wolff; Tina Giner; Valerie Glutsch; Andreas Kerstan; Dagmar Presser; Philipp Schrüfer; Patrick Schummer; Ina Stolze; Judith Weber; Konstantin Drexler; Marion Mickler; Camila Toledo Stauner; Alexander Thiem

doi:10.1016/j.ejca.2019.06.013

Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks

Roman C. Maron, Michael Weichenthal, Jochen S. Utikal, A. Hekler, Carola Berking, Axel Hauschild, Alexander H. Enk, Sebastian Haferkamp, Joachim Klode, Dirk Schadendorf, Philipp Jansen, Tim Holland-Letz, Bastian Schilling, Christof von Kalle, Stefan Fröhling, Maria R. Gaiser, Daniela Hartmann, Anja Gesierich, Katharina C. Kähler, Ulrike WehkampAnte Karoglan, Claudia Bär, Titus J. Brinker, Laurenz Schmitt, Wiebke K. Peitsch, Friederike Hoffmann, Jürgen C. Becker, Christina Drusio, G. Lodde, Stefanie Sammet, Wiebke Sondermann, S. Ugurel, Jeannine Zader, Alexander Enk, Martin Salzmann, S. Schäfer, Knut Schäkel, J. Winkler, Priscilla Wölbing, Hiba Asper, Ann Sophie Bohne, Victoria Brown, Bianca Burba, Sophia Deffaa, Cecilia Dietrich, Matthias Dietrich, Katharina Antonia Drerup, Friederike Egberts, Anna Sophie Erkens, S. Greven, V. Harde, Marion Jost, M. Kaeding, Katharina Kosova, Stephan Lischner, M. Maagk, Anna Laetitia Messinger, M. Metzner, Rogina Motamedi, Ann Christine Rosenthal, Ulrich Seidl, Jana Stemmermann, Kaspar Torz, Juliana Giraldo Velez, Jennifer Haiduk, Mareike Alter, Paul Bergenthal, Anne Gerlach, Christian Holtorf, Sophie Kindermann, L. Kraas, Moritz Felcht, Claus Detlev Klemke, Hjalmar Kurzen, Thomas Leibing, Verena Müller, Raphael R. Reinhard, Jochen Utikal, Franziska Winter, Laurie Eicher, Markus Heppt, Katharina Kilian, Sebastian Krammer, D. Lill, Anne Charlotte Niesert, Eva Oppel, Elke Sattler, S. Senner, Jens Wallmichrath, Hans Wolff, Tina Giner, Valerie Glutsch, Andreas Kerstan, Dagmar Presser, Philipp Schrüfer, Patrick Schummer, Ina Stolze, Judith Weber, Konstantin Drexler, Marion Mickler, Camila Toledo Stauner, Alexander Thiem

Default

Research output: Contribution to journal › Article › peer-review

157 Scopus citations

Abstract

Background: Recently, convolutional neural networks (CNNs) systematically outperformed dermatologists in distinguishing dermoscopic melanoma and nevi images. However, such a binary classification does not reflect the clinical reality of skin cancer screenings in which multiple diagnoses need to be taken into account. Methods: Using 11,444 dermoscopic images, which covered dermatologic diagnoses comprising the majority of commonly pigmented skin lesions commonly faced in skin cancer screenings, a CNN was trained through novel deep learning techniques. A test set of 300 biopsy-verified images was used to compare the classifier's performance with that of 112 dermatologists from 13 German university hospitals. The primary end-point was the correct classification of the different lesions into benign and malignant. The secondary end-point was the correct classification of the images into one of the five diagnostic categories. Findings: Sensitivity and specificity of dermatologists for the primary end-point were 74.4% (95% confidence interval [CI]: 67.0–81.8%) and 59.8% (95% CI: 49.8–69.8%), respectively. At equal sensitivity, the algorithm achieved a specificity of 91.3% (95% CI: 85.5–97.1%). For the secondary end-point, the mean sensitivity and specificity of the dermatologists were at 56.5% (95% CI: 42.8–70.2%) and 89.2% (95% CI: 85.0–93.3%), respectively. At equal sensitivity, the algorithm achieved a specificity of 98.8%. Two-sided McNemar tests revealed significance for the primary end-point (p < 0.001). For the secondary end-point, outperformance (p < 0.001) was achieved except for basal cell carcinoma (on-par performance). Interpretation: Our findings show that automated classification of dermoscopic melanoma and nevi images is extendable to a multiclass classification problem, thus better reflecting clinical differential diagnoses, while still outperforming dermatologists at a significant level (p < 0.001).

Original language	English
Pages (from-to)	57-65
Number of pages	9
Journal	European Journal of Cancer
Volume	119
DOIs	https://doi.org/10.1016/j.ejca.2019.06.013
State	Published - Sep 2019

Keywords

Artificial intelligence
Melanoma
Skin cancer
Skin cancer screening

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1016/j.ejca.2019.06.013

Cite this

Maron, R. C., Weichenthal, M., Utikal, J. S., Hekler, A., Berking, C., Hauschild, A., Enk, A. H., Haferkamp, S., Klode, J., Schadendorf, D., Jansen, P., Holland-Letz, T., Schilling, B., von Kalle, C., Fröhling, S., Gaiser, M. R., Hartmann, D., Gesierich, A., Kähler, K. C., ... Thiem, A. (2019). Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks. European Journal of Cancer, 119, 57-65. https://doi.org/10.1016/j.ejca.2019.06.013

@article{3ad9218057ea4bcbbe0c3cfe50c053d7,

title = "Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks",

abstract = "Background: Recently, convolutional neural networks (CNNs) systematically outperformed dermatologists in distinguishing dermoscopic melanoma and nevi images. However, such a binary classification does not reflect the clinical reality of skin cancer screenings in which multiple diagnoses need to be taken into account. Methods: Using 11,444 dermoscopic images, which covered dermatologic diagnoses comprising the majority of commonly pigmented skin lesions commonly faced in skin cancer screenings, a CNN was trained through novel deep learning techniques. A test set of 300 biopsy-verified images was used to compare the classifier's performance with that of 112 dermatologists from 13 German university hospitals. The primary end-point was the correct classification of the different lesions into benign and malignant. The secondary end-point was the correct classification of the images into one of the five diagnostic categories. Findings: Sensitivity and specificity of dermatologists for the primary end-point were 74.4% (95% confidence interval [CI]: 67.0–81.8%) and 59.8% (95% CI: 49.8–69.8%), respectively. At equal sensitivity, the algorithm achieved a specificity of 91.3% (95% CI: 85.5–97.1%). For the secondary end-point, the mean sensitivity and specificity of the dermatologists were at 56.5% (95% CI: 42.8–70.2%) and 89.2% (95% CI: 85.0–93.3%), respectively. At equal sensitivity, the algorithm achieved a specificity of 98.8%. Two-sided McNemar tests revealed significance for the primary end-point (p < 0.001). For the secondary end-point, outperformance (p < 0.001) was achieved except for basal cell carcinoma (on-par performance). Interpretation: Our findings show that automated classification of dermoscopic melanoma and nevi images is extendable to a multiclass classification problem, thus better reflecting clinical differential diagnoses, while still outperforming dermatologists at a significant level (p < 0.001).",

keywords = "Artificial intelligence, Melanoma, Skin cancer, Skin cancer screening",

author = "Maron, {Roman C.} and Michael Weichenthal and Utikal, {Jochen S.} and A. Hekler and Carola Berking and Axel Hauschild and Enk, {Alexander H.} and Sebastian Haferkamp and Joachim Klode and Dirk Schadendorf and Philipp Jansen and Tim Holland-Letz and Bastian Schilling and {von Kalle}, Christof and Stefan Fr{\"o}hling and Gaiser, {Maria R.} and Daniela Hartmann and Anja Gesierich and K{\"a}hler, {Katharina C.} and Ulrike Wehkamp and Ante Karoglan and Claudia B{\"a}r and Brinker, {Titus J.} and Laurenz Schmitt and Peitsch, {Wiebke K.} and Friederike Hoffmann and Becker, {J{\"u}rgen C.} and Christina Drusio and G. Lodde and Stefanie Sammet and Wiebke Sondermann and S. Ugurel and Jeannine Zader and Alexander Enk and Martin Salzmann and S. Sch{\"a}fer and Knut Sch{\"a}kel and J. Winkler and Priscilla W{\"o}lbing and Hiba Asper and Bohne, {Ann Sophie} and Victoria Brown and Bianca Burba and Sophia Deffaa and Cecilia Dietrich and Matthias Dietrich and Drerup, {Katharina Antonia} and Friederike Egberts and Erkens, {Anna Sophie} and S. Greven and V. Harde and Marion Jost and M. Kaeding and Katharina Kosova and Stephan Lischner and M. Maagk and Messinger, {Anna Laetitia} and M. Metzner and Rogina Motamedi and Rosenthal, {Ann Christine} and Ulrich Seidl and Jana Stemmermann and Kaspar Torz and Velez, {Juliana Giraldo} and Jennifer Haiduk and Mareike Alter and Paul Bergenthal and Anne Gerlach and Christian Holtorf and Sophie Kindermann and L. Kraas and Moritz Felcht and Klemke, {Claus Detlev} and Hjalmar Kurzen and Thomas Leibing and Verena M{\"u}ller and Reinhard, {Raphael R.} and Jochen Utikal and Franziska Winter and Laurie Eicher and Markus Heppt and Katharina Kilian and Sebastian Krammer and D. Lill and Niesert, {Anne Charlotte} and Eva Oppel and Elke Sattler and S. Senner and Jens Wallmichrath and Hans Wolff and Tina Giner and Valerie Glutsch and Andreas Kerstan and Dagmar Presser and Philipp Schr{\"u}fer and Patrick Schummer and Ina Stolze and Judith Weber and Konstantin Drexler and Marion Mickler and Stauner, {Camila Toledo} and Alexander Thiem",

note = "Publisher Copyright: {\textcopyright} 2019 The Author(s)",

year = "2019",

month = sep,

doi = "10.1016/j.ejca.2019.06.013",

language = "English",

volume = "119",

pages = "57--65",

journal = "European Journal of Cancer",

issn = "0959-8049",

publisher = "Elsevier Ltd.",

}

Maron, RC, Weichenthal, M, Utikal, JS, Hekler, A, Berking, C, Hauschild, A, Enk, AH, Haferkamp, S, Klode, J, Schadendorf, D, Jansen, P, Holland-Letz, T, Schilling, B, von Kalle, C, Fröhling, S, Gaiser, MR, Hartmann, D, Gesierich, A, Kähler, KC, Wehkamp, U, Karoglan, A, Bär, C, Brinker, TJ, Schmitt, L, Peitsch, WK, Hoffmann, F, Becker, JC, Drusio, C, Lodde, G, Sammet, S, Sondermann, W, Ugurel, S, Zader, J, Enk, A, Salzmann, M, Schäfer, S, Schäkel, K, Winkler, J, Wölbing, P, Asper, H, Bohne, AS, Brown, V, Burba, B, Deffaa, S, Dietrich, C, Dietrich, M, Drerup, KA, Egberts, F, Erkens, AS, Greven, S, Harde, V, Jost, M, Kaeding, M, Kosova, K, Lischner, S, Maagk, M, Messinger, AL, Metzner, M, Motamedi, R, Rosenthal, AC, Seidl, U, Stemmermann, J, Torz, K, Velez, JG, Haiduk, J, Alter, M, Bergenthal, P, Gerlach, A, Holtorf, C, Kindermann, S, Kraas, L, Felcht, M, Klemke, CD, Kurzen, H, Leibing, T, Müller, V, Reinhard, RR, Utikal, J, Winter, F, Eicher, L, Heppt, M, Kilian, K, Krammer, S, Lill, D, Niesert, AC, Oppel, E, Sattler, E, Senner, S, Wallmichrath, J, Wolff, H, Giner, T, Glutsch, V, Kerstan, A, Presser, D, Schrüfer, P, Schummer, P, Stolze, I, Weber, J, Drexler, K, Mickler, M, Stauner, CT & Thiem, A 2019, 'Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks', European Journal of Cancer, vol. 119, pp. 57-65. https://doi.org/10.1016/j.ejca.2019.06.013

TY - JOUR

T1 - Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks

AU - Maron, Roman C.

AU - Weichenthal, Michael

AU - Utikal, Jochen S.

AU - Hekler, A.

AU - Berking, Carola

AU - Hauschild, Axel

AU - Enk, Alexander H.

AU - Haferkamp, Sebastian

AU - Klode, Joachim

AU - Schadendorf, Dirk

AU - Jansen, Philipp

AU - Holland-Letz, Tim

AU - Schilling, Bastian

AU - von Kalle, Christof

AU - Fröhling, Stefan

AU - Gaiser, Maria R.

AU - Hartmann, Daniela

AU - Gesierich, Anja

AU - Kähler, Katharina C.

AU - Wehkamp, Ulrike

AU - Karoglan, Ante

AU - Bär, Claudia

AU - Brinker, Titus J.

AU - Schmitt, Laurenz

AU - Peitsch, Wiebke K.

AU - Hoffmann, Friederike

AU - Becker, Jürgen C.

AU - Drusio, Christina

AU - Lodde, G.

AU - Sammet, Stefanie

AU - Sondermann, Wiebke

AU - Ugurel, S.

AU - Zader, Jeannine

AU - Enk, Alexander

AU - Salzmann, Martin

AU - Schäfer, S.

AU - Schäkel, Knut

AU - Winkler, J.

AU - Wölbing, Priscilla

AU - Asper, Hiba

AU - Bohne, Ann Sophie

AU - Brown, Victoria

AU - Burba, Bianca

AU - Deffaa, Sophia

AU - Dietrich, Cecilia

AU - Dietrich, Matthias

AU - Drerup, Katharina Antonia

AU - Egberts, Friederike

AU - Erkens, Anna Sophie

AU - Greven, S.

AU - Harde, V.

AU - Jost, Marion

AU - Kaeding, M.

AU - Kosova, Katharina

AU - Lischner, Stephan

AU - Maagk, M.

AU - Messinger, Anna Laetitia

AU - Metzner, M.

AU - Motamedi, Rogina

AU - Rosenthal, Ann Christine

AU - Seidl, Ulrich

AU - Stemmermann, Jana

AU - Torz, Kaspar

AU - Velez, Juliana Giraldo

AU - Haiduk, Jennifer

AU - Alter, Mareike

AU - Bergenthal, Paul

AU - Gerlach, Anne

AU - Holtorf, Christian

AU - Kindermann, Sophie

AU - Kraas, L.

AU - Felcht, Moritz

AU - Klemke, Claus Detlev

AU - Kurzen, Hjalmar

AU - Leibing, Thomas

AU - Müller, Verena

AU - Reinhard, Raphael R.

AU - Utikal, Jochen

AU - Winter, Franziska

AU - Eicher, Laurie

AU - Heppt, Markus

AU - Kilian, Katharina

AU - Krammer, Sebastian

AU - Lill, D.

AU - Niesert, Anne Charlotte

AU - Oppel, Eva

AU - Sattler, Elke

AU - Senner, S.

AU - Wallmichrath, Jens

AU - Wolff, Hans

AU - Giner, Tina

AU - Glutsch, Valerie

AU - Kerstan, Andreas

AU - Presser, Dagmar

AU - Schrüfer, Philipp

AU - Schummer, Patrick

AU - Stolze, Ina

AU - Weber, Judith

AU - Drexler, Konstantin

AU - Mickler, Marion

AU - Stauner, Camila Toledo

AU - Thiem, Alexander

PY - 2019/9

Y1 - 2019/9

N2 - Background: Recently, convolutional neural networks (CNNs) systematically outperformed dermatologists in distinguishing dermoscopic melanoma and nevi images. However, such a binary classification does not reflect the clinical reality of skin cancer screenings in which multiple diagnoses need to be taken into account. Methods: Using 11,444 dermoscopic images, which covered dermatologic diagnoses comprising the majority of commonly pigmented skin lesions commonly faced in skin cancer screenings, a CNN was trained through novel deep learning techniques. A test set of 300 biopsy-verified images was used to compare the classifier's performance with that of 112 dermatologists from 13 German university hospitals. The primary end-point was the correct classification of the different lesions into benign and malignant. The secondary end-point was the correct classification of the images into one of the five diagnostic categories. Findings: Sensitivity and specificity of dermatologists for the primary end-point were 74.4% (95% confidence interval [CI]: 67.0–81.8%) and 59.8% (95% CI: 49.8–69.8%), respectively. At equal sensitivity, the algorithm achieved a specificity of 91.3% (95% CI: 85.5–97.1%). For the secondary end-point, the mean sensitivity and specificity of the dermatologists were at 56.5% (95% CI: 42.8–70.2%) and 89.2% (95% CI: 85.0–93.3%), respectively. At equal sensitivity, the algorithm achieved a specificity of 98.8%. Two-sided McNemar tests revealed significance for the primary end-point (p < 0.001). For the secondary end-point, outperformance (p < 0.001) was achieved except for basal cell carcinoma (on-par performance). Interpretation: Our findings show that automated classification of dermoscopic melanoma and nevi images is extendable to a multiclass classification problem, thus better reflecting clinical differential diagnoses, while still outperforming dermatologists at a significant level (p < 0.001).

AB - Background: Recently, convolutional neural networks (CNNs) systematically outperformed dermatologists in distinguishing dermoscopic melanoma and nevi images. However, such a binary classification does not reflect the clinical reality of skin cancer screenings in which multiple diagnoses need to be taken into account. Methods: Using 11,444 dermoscopic images, which covered dermatologic diagnoses comprising the majority of commonly pigmented skin lesions commonly faced in skin cancer screenings, a CNN was trained through novel deep learning techniques. A test set of 300 biopsy-verified images was used to compare the classifier's performance with that of 112 dermatologists from 13 German university hospitals. The primary end-point was the correct classification of the different lesions into benign and malignant. The secondary end-point was the correct classification of the images into one of the five diagnostic categories. Findings: Sensitivity and specificity of dermatologists for the primary end-point were 74.4% (95% confidence interval [CI]: 67.0–81.8%) and 59.8% (95% CI: 49.8–69.8%), respectively. At equal sensitivity, the algorithm achieved a specificity of 91.3% (95% CI: 85.5–97.1%). For the secondary end-point, the mean sensitivity and specificity of the dermatologists were at 56.5% (95% CI: 42.8–70.2%) and 89.2% (95% CI: 85.0–93.3%), respectively. At equal sensitivity, the algorithm achieved a specificity of 98.8%. Two-sided McNemar tests revealed significance for the primary end-point (p < 0.001). For the secondary end-point, outperformance (p < 0.001) was achieved except for basal cell carcinoma (on-par performance). Interpretation: Our findings show that automated classification of dermoscopic melanoma and nevi images is extendable to a multiclass classification problem, thus better reflecting clinical differential diagnoses, while still outperforming dermatologists at a significant level (p < 0.001).

KW - Artificial intelligence

KW - Melanoma

KW - Skin cancer

KW - Skin cancer screening

UR - http://www.scopus.com/inward/record.url?scp=85069038904&partnerID=8YFLogxK

U2 - 10.1016/j.ejca.2019.06.013

DO - 10.1016/j.ejca.2019.06.013

M3 - Article

C2 - 31419752

AN - SCOPUS:85069038904

SN - 0959-8049

VL - 119

SP - 57

EP - 65

JO - European Journal of Cancer

JF - European Journal of Cancer

ER -

Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks

Abstract

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this