TY - JOUR

T1 - Transformations establishing equivalence across neural networks

T2 - When have two networks learned the same task?

AU - Bertalan, Tom

AU - Dietrich, Felix

AU - Kevrekidis, Ioannis G.

N1 - Publisher Copyright:
© 2024 Author(s).

PY - 2024/7/1

Y1 - 2024/7/1

N2 - Transformations are a key tool in the qualitative study of dynamical systems: transformations to a normal form, for example, underpin the study of instabilities and bifurcations. In this work, we test, and when possible establish, an equivalence between two different artificial neural networks by attempting to construct a data-driven transformation between them, using diffusion maps with a Mahalanobis-like metric. If the construction succeeds, the two networks can be thought of as belonging to the same equivalence class. We first discuss transformation functions between only the outputs of the two networks; we then also consider transformations that take into account outputs (activations) of a number of internal neurons from each network. Whitney’s theorem dictates the number of (generic) measurements from one of the networks required to reconstruct each and every feature of the second network. The construction of the transformation function relies on a consistent, intrinsic representation of the network input space. We illustrate our algorithm by matching neural network pairs trained to learn (a) observations of scalar functions, (b) observations of two-dimensional vector fields, and (c) representations of images of a moving three-dimensional object (a rotating horse). We also demonstrate reconstruction of a network’s input (and output) from minimal partial observations of intermediate neuron activations. The construction of equivalences across different network instantiations clearly relates to transfer learning and will also be valuable in establishing equivalence between different machine learning-based tools.

AB - Transformations are a key tool in the qualitative study of dynamical systems: transformations to a normal form, for example, underpin the study of instabilities and bifurcations. In this work, we test, and when possible establish, an equivalence between two different artificial neural networks by attempting to construct a data-driven transformation between them, using diffusion maps with a Mahalanobis-like metric. If the construction succeeds, the two networks can be thought of as belonging to the same equivalence class. We first discuss transformation functions between only the outputs of the two networks; we then also consider transformations that take into account outputs (activations) of a number of internal neurons from each network. Whitney’s theorem dictates the number of (generic) measurements from one of the networks required to reconstruct each and every feature of the second network. The construction of the transformation function relies on a consistent, intrinsic representation of the network input space. We illustrate our algorithm by matching neural network pairs trained to learn (a) observations of scalar functions, (b) observations of two-dimensional vector fields, and (c) representations of images of a moving three-dimensional object (a rotating horse). We also demonstrate reconstruction of a network’s input (and output) from minimal partial observations of intermediate neuron activations. The construction of equivalences across different network instantiations clearly relates to transfer learning and will also be valuable in establishing equivalence between different machine learning-based tools.

UR - http://www.scopus.com/inward/record.url?scp=85199126567&partnerID=8YFLogxK

U2 - 10.1063/5.0206406

DO - 10.1063/5.0206406

M3 - Article

C2 - 39023924

AN - SCOPUS:85199126567

SN - 1054-1500

VL - 34

JO - Chaos

JF - Chaos

IS - 7

M1 - 073138

ER -