Automatic Evaluation of Automatically Derived Semantic Scenario Instance Descriptions

Scenario-based testing is a promising approach for testing autonomous driving systems at the system-level. In this work, we focus on concrete scenario instances. These can be extracted from recorded traffic data. However, to make the recordings usable in the scenario-based testing approach, we need to enrich the originally uninformative data with meaning-ful descriptions. This way we obtain semantic scenario instance descriptions that are necessary for subsequent steps of scenario-based testing. Several approaches attempt to overcome the manual effort to derive these descriptions and aim to automate it. However, most of the works lack a systematic evaluation of the quality of the derived descriptions and, therefore, do not solve the problem of manual effort entirely. In this work, we address this shortcoming by providing an approach that enables a context-independent, automated, systematic evaluation of the derived semantic scenario instance descriptions of arbitrary scenario types. Further, we present how not only the artifacts, but also the approach used for deriving the descriptions itself can be evaluated. We demonstrate the applicability of the presented evaluation on an intersection dataset.