Annotator trustability-based cooperative learning solutions for intelligent audio analysis

A broad range of artificially intelligent applications are nowadays available resulting in a need for masses of labelled data for the underlying machine learning models. This annotated data, however, is scarce and expensive to obtain from expert-like annotators. Crowdsourcing has been shown as a viable alternative, but it has to be carried out with adequate quality control to obtain reliable labels. Whilst crowdsourcing allows for the rapid collection of large-scale annotations, another technique called Cooperative Learning, aims at reducing the overall annotation costs, by learning to select only the most important instances for manual annotation. In this regard, we investigate the advantages of this approach and combine crowdsourcing with different iterative cooperative learning paradigms for audio data annotation, incorporating an annotator trustability score to reduce the labelling effort needed and, at the same time, to achieve better classification results. Key experimental results on an emotion recognition task show a considerable relative annotation reduction compared to a 'non-intelligent' approach of up to 85.3 %. Moreover, the proposed trustability-based methods reach an unweighted average recall of 74.8 %, while the baseline approach peaks at 61.2 %. Therefore, the proposed trustability-based approaches efficiently reduce the manual annotation load, as well as improving the model.