Approaching Peak Ground Truth

Florian Kofler, Johannes Wahle, Ivan Ezhov, Sophia J. Wagner, Rami Al-Maskari, Emilia Gryska, Mihail Todorov, Christina Bukas, Felix Meissen, Tingying Peng, Ali Erturk, Daniel Rueckert, Rolf Heckemann, Jan Kirschke, Claus Zimmer, Benedikt Wiestler, Bjoern Menze, Marie Piraud

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Machine learning models are typically evaluated by computing similarity with reference annotations and trained by maximizing similarity with such. Especially in the biomedical domain, annotations are subjective and suffer from low inter-and intra-rater reliability. Since annotations only reflect one interpretation of the real world, this can lead to sub-optimal predictions even though the model achieves high similarity scores. Here, the theoretical concept of Peak Ground Truth (PGT) is introduced. PGT marks the point beyond which an increase in similarity with the reference annotation stops translating to better Real World Model Performance (RWMP). Additionally, a quantitative technique to approximate PGT by computing inter- and intra-rater reliability is proposed. Finally, four categories of PGT-aware strategies to evaluate and improve model performance are reviewed.

Original languageEnglish
Title of host publication2023 IEEE International Symposium on Biomedical Imaging, ISBI 2023
PublisherIEEE Computer Society
ISBN (Electronic)9781665473583
DOIs
StatePublished - 2023
Event20th IEEE International Symposium on Biomedical Imaging, ISBI 2023 - Cartagena, Colombia
Duration: 18 Apr 202321 Apr 2023

Publication series

NameProceedings - International Symposium on Biomedical Imaging
Volume2023-April
ISSN (Print)1945-7928
ISSN (Electronic)1945-8452

Conference

Conference20th IEEE International Symposium on Biomedical Imaging, ISBI 2023
Country/TerritoryColombia
CityCartagena
Period18/04/2321/04/23

Keywords

  • annotation
  • deep learning
  • ground truth
  • machine learning
  • reference
  • segmentation

Fingerprint

Dive into the research topics of 'Approaching Peak Ground Truth'. Together they form a unique fingerprint.

Cite this