JOINT OPTIMIZATION OF MULTIPLE NEURAL CODEBOOKS IN A HYBRID CONNECTIONIST-HMM SPEECH RECOGNITION SYSTEM

This paper proposes a new approach for a hybrid connectionistHMM speech recognition system. The system consists of a multi-feature HMM-based recognition module using three different neural networks as multiple neural codebooks. Each neural network receives a different feature (i.e. cepstrum, delta cepstrum, and delta power) as input and generates a vector quantizer label obtained from the firing neuron in the output layer. The neural networks are first trained separately using a special self-organizing information theory-based learning method. A 26% error reduction is obtained with this method, compared to the performance of the same system using multiple k-means vector quantizers with the same codebook size. In a second training phase, the neural codebooks are further refined by extending the information theory-based training criterion into a joint criterion reflecting the joint information content and the dependencies of the three different label streams. This further improves the error reduction rate to 30%.