COMPUTER VALIDATION OF NEURAL NETWORK DYNAMICS: A FIRST CASE STUDY

A large number of current machine learning methods rely upon deep neural networks. Yet, viewing neural networks as nonlinear dynamical systems, it becomes quickly apparent that mathematically rigorously establishing certain patterns generated by the nodes in the network is extremely difficult. Indeed, it is well-understood in the nonlinear dynamics of complex systems that, even in low-dimensional models, analytical techniques rooted in pencil-and-paper approaches frequently reach their limits. In this work, we proposed a completely different perspective via the paradigm of validated numerical methods of nonlinear dynamics. The idea was to use computer-assisted proofs to validate mathematically the existence of nonlinear patterns in neural networks. As a case study, we considered a class of recurrent neural networks, where we proved via computer assistance the existence of several hundred Hopf bifurcation points, their non-degeneracy, and, hence, also the existence of several hundred periodic orbits. Our paradigm has the capability to rigorously verify complex nonlinear behavior of neural networks, which provided a first step to explain the full abilities, as well as potential sensitivities, of machine learning methods via computer-assisted proofs. We showcased how validated numerical techniques can shed light on the internal working of recurrent neural networks (RNNs). For this, proofs of Hopf bifurcations were a first step toward an integration of dynamical system theory in practical application of RNNs by proving the existence of periodic orbits in a variety of settings.