Towards evidence-based computational statistics: Lessons from clinical research on the role and design of real-data benchmark studies

Anne Laure Boulesteix, Rory Wilson, Alexander Hapfelmeier

Research output: Contribution to journalArticlepeer-review

38 Scopus citations


Background: The goal of medical research is to develop interventions that are in some sense superior, with respect to patient outcome, to interventions currently in use. Similarly, the goal of research in methodological computational statistics is to develop data analysis tools that are themselves superior to the existing tools. The methodology of the evaluation of medical interventions continues to be discussed extensively in the literature and it is now well accepted that medicine should be at least partly "evidence-based". Although we statisticians are convinced of the importance of unbiased, well-thought-out study designs and evidence-based approaches in the context of clinical research, we tend to ignore these principles when designing our own studies for evaluating statistical methods in the context of our methodological research. Main message: In this paper, we draw an analogy between clinical trials and real-data-based benchmarking experiments in methodological statistical science, with datasets playing the role of patients and methods playing the role of medical interventions. Through this analogy, we suggest directions for improvement in the design and interpretation of studies which use real data to evaluate statistical methods, in particular with respect to dataset inclusion criteria and the reduction of various forms of bias. More generally, we discuss the concept of "evidence-based" statistical research, its limitations and its impact on the design and interpretation of real-data-based benchmark experiments. Conclusion: We suggest that benchmark studies - a method of assessment of statistical methods using real-world datasets - might benefit from adopting (some) concepts from evidence-based medicine towards the goal of more evidence-based statistical research.

Original languageEnglish
Article number138
JournalBMC Medical Research Methodology
Issue number1
StatePublished - 9 Sep 2017
Externally publishedYes


  • Clinical trial
  • Comparison study
  • Good practice
  • Method evaluation


Dive into the research topics of 'Towards evidence-based computational statistics: Lessons from clinical research on the role and design of real-data benchmark studies'. Together they form a unique fingerprint.

Cite this