Towards a Comprehensive Evaluation of Decision Rules and Decision Mining Algorithms Beyond Accuracy

Decision mining algorithms discover decision points and the corresponding decision rules in business processes. So far, the evaluation of decision mining algorithms has focused on performance (e.g., accuracy), neglecting the impact of other criteria, e.g., understandability or consistency of the discovered decision model. However, performance alone cannot reflect if the discovered decision rules produce value to the user by providing insights into the process. Providing metrics to comprehensively evaluate the decision model and decision rules can lead to more meaningful insights and assessment of decision mining algorithms. In this paper, we examine the ability of different criteria from software engineering, explainable AI, and process mining that go beyond performance to evaluate decision mining results and propose metrics to measure these criteria. To evaluate the proposed metrics, they are applied to different decision algorithms on two synthetic and one real-life dataset. The results are compared to the findings of a user study to check whether they align with user perception. As a result, we suggest four metrics that enable a comprehensive evaluation of decision mining results and a more in-depth comparison of different decision mining algorithms. In addition, guidelines for formulating decision rules are presented.