Initial state training procedure improves dynamic recurrent networks with time-dependent weights

The problem of learning multiple continuous trajectories by means of recurrent neural networks with (in general) time-varying weights is addressed in this study. The learning process is transformed into an optimal control framework where both the weights and the initial network state to be found are treated as controls. For such a task, a new learning algorithm is proposed which is based on a variational formulation of Pontryagin's maximum principle. The convergence of this algorithm, under reasonable assumptions, is also investigated. Numerical examples of learning nontrivial two-class problems are presented which demonstrate the efficiency of the approach proposed.