TY - GEN
T1 - Advancing On-Device Neural Network Training with TinyPropv2
T2 - 2024 International Joint Conference on Neural Networks, IJCNN 2024
AU - Rüb, Marcus
AU - Sikora, Axel
AU - Mueller-Gritschneder, Daniel
N1 - Publisher Copyright:
© 2024 IEEE.
PY - 2024
Y1 - 2024
N2 - This study introduces EmbeddedTrain, an innovative algorithm optimized for on-device learning in deep neural networks, specifically designed for low-power microcontroller units. EmbeddedTrain refines sparse backpropagation by dynamically adjusting the level of sparity, including the ability to selectively skip training steps. This feature significantly lowers computational effort without substantially compromising accuracy. Our comprehensive evaluation across diverse datasets - CIFAR 10, CIFAR100, Flower, Food, Speech Command, MNIST, HAR, and DCASE2020 - reveals that EmbeddedTrain achieves near-parity with full training methods, with an average accuracy drop of only around 1% in most cases. For instance, against full training, EmbeddedTrain's accuracy drop is minimal, for example, only 0.82% on CIFAR 10 and 1.07% on CIFAR100. In terms of computational effort, EmbeddedTrain shows a marked reduction, requiring as little as 10% of the computational effort needed for full training in some scenarios, and consistently outperforms other sparse training methodologies. These findings underscore EmbeddedTrain's capacity to efficiently manage computational resources while maintaining high accuracy, positioning it as an advantageous solution for advanced embedded device applications in the IoT ecosystem.
AB - This study introduces EmbeddedTrain, an innovative algorithm optimized for on-device learning in deep neural networks, specifically designed for low-power microcontroller units. EmbeddedTrain refines sparse backpropagation by dynamically adjusting the level of sparity, including the ability to selectively skip training steps. This feature significantly lowers computational effort without substantially compromising accuracy. Our comprehensive evaluation across diverse datasets - CIFAR 10, CIFAR100, Flower, Food, Speech Command, MNIST, HAR, and DCASE2020 - reveals that EmbeddedTrain achieves near-parity with full training methods, with an average accuracy drop of only around 1% in most cases. For instance, against full training, EmbeddedTrain's accuracy drop is minimal, for example, only 0.82% on CIFAR 10 and 1.07% on CIFAR100. In terms of computational effort, EmbeddedTrain shows a marked reduction, requiring as little as 10% of the computational effort needed for full training in some scenarios, and consistently outperforms other sparse training methodologies. These findings underscore EmbeddedTrain's capacity to efficiently manage computational resources while maintaining high accuracy, positioning it as an advantageous solution for advanced embedded device applications in the IoT ecosystem.
UR - http://www.scopus.com/inward/record.url?scp=85204954696&partnerID=8YFLogxK
U2 - 10.1109/IJCNN60899.2024.10650122
DO - 10.1109/IJCNN60899.2024.10650122
M3 - Conference contribution
AN - SCOPUS:85204954696
T3 - Proceedings of the International Joint Conference on Neural Networks
BT - 2024 International Joint Conference on Neural Networks, IJCNN 2024 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 30 June 2024 through 5 July 2024
ER -