TY - GEN
T1 - Federated Fine-Tuning of LLMs on the Very Edge
T2 - 8th Workshop on Data Management for End-to-End Machine Learning, DEEM 2024
AU - Woisetschläger, Herbert
AU - Erben, Alexander
AU - Wang, Shiqiang
AU - Mayer, Ruben
AU - Jacobsen, Hans Arno
N1 - Publisher Copyright:
© 2024 ACM.
PY - 2024/6/9
Y1 - 2024/6/9
N2 - With the emergence of AI regulations, such as the EU AI Act, requirements for simple data lineage, enforcement of low data bias, and energy efficiency have become a priority for everyone offering AI services. Being pre-trained on versatile and a vast amount of data, large language models and foundation models (FMs) offer a good basis for building high-quality deep learning pipelines. Fine-tuning can further improve model performance on a specific downstream task, which requires orders of magnitude less data than pre-training. Often, access to high-quality and low-bias data for model fine-tuning is limited due to technical or regulatory requirements. Federated learning (FL), as a distributed and privacy-preserving technique, offers a well-suited approach to significantly expanding data access for model fine-tuning. Yet, this data is often located on the network edge, where energy, computational, and communication resources are significantly more limited than in data centers. In our paper, we conduct an end-to-end evaluation for fine-tuning the FLAN-T5 FM family on the network edge. We study energy efficiency potentials throughout FL systems - on clients, in communication, and on the server. Our analysis introduces energy efficiency as a real-time metric to assess the computational efficiency of an FL system. We show the stark need for further improvements in communication efficiency when working with FMs and demonstrate the importance of adaptive FL optimizers for FM training.
AB - With the emergence of AI regulations, such as the EU AI Act, requirements for simple data lineage, enforcement of low data bias, and energy efficiency have become a priority for everyone offering AI services. Being pre-trained on versatile and a vast amount of data, large language models and foundation models (FMs) offer a good basis for building high-quality deep learning pipelines. Fine-tuning can further improve model performance on a specific downstream task, which requires orders of magnitude less data than pre-training. Often, access to high-quality and low-bias data for model fine-tuning is limited due to technical or regulatory requirements. Federated learning (FL), as a distributed and privacy-preserving technique, offers a well-suited approach to significantly expanding data access for model fine-tuning. Yet, this data is often located on the network edge, where energy, computational, and communication resources are significantly more limited than in data centers. In our paper, we conduct an end-to-end evaluation for fine-tuning the FLAN-T5 FM family on the network edge. We study energy efficiency potentials throughout FL systems - on clients, in communication, and on the server. Our analysis introduces energy efficiency as a real-time metric to assess the computational efficiency of an FL system. We show the stark need for further improvements in communication efficiency when working with FMs and demonstrate the importance of adaptive FL optimizers for FM training.
UR - http://www.scopus.com/inward/record.url?scp=85196629786&partnerID=8YFLogxK
U2 - 10.1145/3650203.3663331
DO - 10.1145/3650203.3663331
M3 - Conference contribution
AN - SCOPUS:85196629786
T3 - Proceedings of the 8th Workshop on Data Management for End-to-End Machine Learning, DEEM 2024 - In conjunction with the 2024 ACM SIGMOD/PODS Conference
SP - 39
EP - 50
BT - Proceedings of the 8th Workshop on Data Management for End-to-End Machine Learning, DEEM 2024 - In conjunction with the 2024 ACM SIGMOD/PODS Conference
PB - Association for Computing Machinery, Inc
Y2 - 9 June 2024
ER -