Two-stage speaker adaptation of hybrid tied-posterior acoustic models

For Gaussian distribution-acoustic models there exist many established technologies for speaker adaptation. Contrary to that, there are almost no well-functioning adaptation methods for hybrid systems, consisting of a combination of HMMs and neural networks. In this paper, strategies are explored to adapt hybrid NN/HMM systems based on the tied-posterior paradigm. We investigate the retraining of selected important parts of the neural network and a gradient based adaptation strategy for the HMM's mixture co-efficients based on maximizing the scaled likelihood. The paper presents the following innovations: First it introduces one of the first adaptation methods for hybrid systems where the HMM component contributes significantly to the adaptation success. Second, it presents a novel approach to the neural network's adaptation, based on the selection of suitable neurons for adaptation. Results on the WSJ speaker adaptation test show the capability of our methods to adapt to new speakers especially in case of adapting the neural net and that both methods can be combined to achieve additional improvement of the word error rate in most cases.