TY - GEN
T1 - Measuring the effect of a guideline-based training on ontology design with a competency questions based evaluation approach
AU - Boeker, Martin
AU - Grewe, Niels
AU - Röhl, Johannes
AU - Schober, Daniel
AU - Schulz, Stefan
AU - Seddig-Raufie, Djamila
AU - Jansen, Ludger
N1 - Publisher Copyright:
© 2013 Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI).
PY - 2013
Y1 - 2013
N2 - OBJECTIVE: (a) To measure the effect of a guideline-based training on the performance of ontology developers compared with the performance after unspecific training by a competency question based evaluation; and (b) to provide empirical evidence for the applicability of competency questions in formal ontology evaluation in general. BACKGROUND: A close connection between ontology development and ontology evaluation as quality management procedure can been attained with the use of competency questions. Competency questions are often used as a semi-formal specification of requirements for an ontology under development. Hence they can also be used as evaluation instruments, in order to check how far an ontology fulfills these requirements. METHODS: A randomized controlled trial was conducted with two groups of 12 students each. The intervention consisted in a differential guideline-based training on ontology development vs. unspecific training. After a group-specific training focusing on three topics per group, performance of students was assessed with 12 exercises (2 exercises per topic), in which the students had to apply their skills. Different types of competency questions were elaborated for the analysis of the students' ontologies. We used the proportion of correct answers as a measure of ontology quality. RESULTS: On single topic level, the performance of ontology developers increased after guideline-based training for two out of the six topics: it increased from a proportion of 0.46 to 0.63 for the topic Process & Participation and from 0.44 to 0.53 for the topic Collective Material Entity. In regression analysis, a positive correlation was shown between the performance of students on untrained topics and the performance after specific guideline-based training. Moreover, in multiple regression analysis an overall effect of specific training of 0.09 was calculated (p < 0.1). CONCLUSION: The results show an effect of a specific guideline-based training on the performance of ontology developers compared to the performance after unspecific training by an increase of about 10 % on the rate of correct competency questions. In addition, this study has shown the general applicability of competency questions in a formal ontology evaluation scenario. However, the study also shows that the training of ontology developers and their performance evaluation is a tedious task. The resulting performance of ontology developers is more dependent on the a priori individual competencies than on the specific acquired skills after training.
AB - OBJECTIVE: (a) To measure the effect of a guideline-based training on the performance of ontology developers compared with the performance after unspecific training by a competency question based evaluation; and (b) to provide empirical evidence for the applicability of competency questions in formal ontology evaluation in general. BACKGROUND: A close connection between ontology development and ontology evaluation as quality management procedure can been attained with the use of competency questions. Competency questions are often used as a semi-formal specification of requirements for an ontology under development. Hence they can also be used as evaluation instruments, in order to check how far an ontology fulfills these requirements. METHODS: A randomized controlled trial was conducted with two groups of 12 students each. The intervention consisted in a differential guideline-based training on ontology development vs. unspecific training. After a group-specific training focusing on three topics per group, performance of students was assessed with 12 exercises (2 exercises per topic), in which the students had to apply their skills. Different types of competency questions were elaborated for the analysis of the students' ontologies. We used the proportion of correct answers as a measure of ontology quality. RESULTS: On single topic level, the performance of ontology developers increased after guideline-based training for two out of the six topics: it increased from a proportion of 0.46 to 0.63 for the topic Process & Participation and from 0.44 to 0.53 for the topic Collective Material Entity. In regression analysis, a positive correlation was shown between the performance of students on untrained topics and the performance after specific guideline-based training. Moreover, in multiple regression analysis an overall effect of specific training of 0.09 was calculated (p < 0.1). CONCLUSION: The results show an effect of a specific guideline-based training on the performance of ontology developers compared to the performance after unspecific training by an increase of about 10 % on the rate of correct competency questions. In addition, this study has shown the general applicability of competency questions in a formal ontology evaluation scenario. However, the study also shows that the training of ontology developers and their performance evaluation is a tedious task. The resulting performance of ontology developers is more dependent on the a priori individual competencies than on the specific acquired skills after training.
UR - http://www.scopus.com/inward/record.url?scp=84940266545&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:84940266545
T3 - Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI)
SP - 1783
EP - 1795
BT - Informatik 2013 - 43. Jahrestagung der Gesellschaft fur Informatik e.V. (GI), Informatik Angepasst an Mensch, Organisation und Umwelt
A2 - Horbach, Matthias
PB - Gesellschaft fur Informatik (GI)
T2 - 43. Jahrestagung der Gesellschaft fur Informatik e.V. ,GI, Informatik Angepasst an Mensch, Organisation und Umwelt, Informatik 2013 - 43rd Annual Meeting of the German Informatics Society (GI), Computer Science Adapting to Individuals, Organizations and Environment, Informatics 2013
Y2 - 16 September 2013 through 20 September 2013
ER -