TY - JOUR
T1 - Stable recovery of entangled weights
T2 - Towards robust identification of deep neural networks from minimal samples
AU - Fiedler, Christian
AU - Fornasier, Massimo
AU - Klock, Timo
AU - Rauchensteiner, Michael
N1 - Publisher Copyright:
© 2022 Elsevier Inc.
PY - 2023/1
Y1 - 2023/1
N2 - In this paper we approach the problem of unique and stable identifiability from a finite number of input-output samples of generic feedforward deep artificial neural networks of prescribed architecture with pyramidal shape up to the penultimate layer and smooth activation functions. More specifically we introduce the so-called entangled weights, which compose weights of successive layers intertwined with suitable diagonal and invertible matrices depending on the activation functions and their shifts. We prove that instances of entangled weights are completely and stably approximated by an efficient and robust algorithm as soon as O(D2×m) nonadaptive input-output samples of the network are collected, where D is the input dimension and m is the number of neurons of the network. Moreover, we empirically observe that the approach applies to networks with up to O(D×mL) neurons, where mL is the number of output neurons at layer L. Provided knowledge of layer assignments of entangled weights and of remaining scaling and shift parameters, which may be further heuristically obtained by least squares fitting, the entangled weights identify the network completely and uniquely. To highlight the relevance of the theoretical result of stable recovery of entangled weights, we present numerical experiments, which demonstrate that multilayered networks with generic weights can be robustly identified and therefore uniformly approximated by the presented algorithmic pipeline.
AB - In this paper we approach the problem of unique and stable identifiability from a finite number of input-output samples of generic feedforward deep artificial neural networks of prescribed architecture with pyramidal shape up to the penultimate layer and smooth activation functions. More specifically we introduce the so-called entangled weights, which compose weights of successive layers intertwined with suitable diagonal and invertible matrices depending on the activation functions and their shifts. We prove that instances of entangled weights are completely and stably approximated by an efficient and robust algorithm as soon as O(D2×m) nonadaptive input-output samples of the network are collected, where D is the input dimension and m is the number of neurons of the network. Moreover, we empirically observe that the approach applies to networks with up to O(D×mL) neurons, where mL is the number of output neurons at layer L. Provided knowledge of layer assignments of entangled weights and of remaining scaling and shift parameters, which may be further heuristically obtained by least squares fitting, the entangled weights identify the network completely and uniquely. To highlight the relevance of the theoretical result of stable recovery of entangled weights, we present numerical experiments, which demonstrate that multilayered networks with generic weights can be robustly identified and therefore uniformly approximated by the presented algorithmic pipeline.
KW - Active sampling
KW - Deep neural networks
KW - Deparametrization
KW - Exact finite sample identifiability
KW - Frames
KW - Nonconvex optimization on matrix spaces
UR - http://www.scopus.com/inward/record.url?scp=85137275660&partnerID=8YFLogxK
U2 - 10.1016/j.acha.2022.08.002
DO - 10.1016/j.acha.2022.08.002
M3 - Article
AN - SCOPUS:85137275660
SN - 1063-5203
VL - 62
SP - 123
EP - 172
JO - Applied and Computational Harmonic Analysis
JF - Applied and Computational Harmonic Analysis
ER -