Selecting system architecture: What a single industrial experiment can tell us about the traps to avoid when choosing selection criteria

Decisions related to system architecture are difficult because of fuzziness and lack of information combined with often-conflicting objectives. We organized an industrial workshop with the objective of choosing 5 out of 800 architectures. The first step, the identification of selection criteria, proved to be the greatest challenge. As a result, designers selected system architectures that did not satisfy them without being able to explain why. It appeared that most of the difficulties faced by the designers came from the criteria used for architecture selection. This study aims to identify what made the selection criteria difficult to use. The audio recordings of the workshop were transcribed and analyzed in order to identify the obstacles related to the definition and the use of selection criteria. The analysis highlights two issues: the interdisciplinarity of system architecture makes criteria interdependent and the lack of information makes it impossible to define an exhaustive set of criteria. Finally, this study provides recommendations for selecting appropriate selection criteria and insights for future selection support tools dedicated to system architecture design.