TY - GEN
T1 - Integration of apache spark with invasive resource manager
AU - Chacko, Jeeta Ann
AU - Ureña, Isaías A.Comprés
AU - Gerndt, Michael
N1 - Publisher Copyright:
© 2019 IEEE.
PY - 2019/8
Y1 - 2019/8
N2 - The scheduling of resources on High-Performance Computing systems (HPC) for compute-intensive scientific applications often results in idle nodes at different points in time due to the difference in node requirements for each application. One option to optimize node usage is to assign all idle nodes to data analytics applications. Scientific applications generate data as output which can be used as input for data analytics applications. So it would be beneficial if both types of applications can be run on the same HPC system. The Invasive Resource Manager (IRM) is an extension of the Simple Linux Utility Resource Manager (SLURM) with dynamic resource management and scheduling capabilities on HPC systems and Apache Spark is an open-source cluster computing framework that is widely used for data analytics applications. This project integrates Apache Spark with the IRM so that data analytics applications can be run on HPC systems with dynamic resource allocation. This work also collects performance data from Spark applications and improves the existing scheduling strategy of the IRM. The integrated system is deployed on SuperMUC, the supercomputer at the Leibniz Supercomputing Centre in Germany for testing and evaluation. This project illustrates the design for integrating data analytics on HPC systems with the additional advantage of improving resource utilization. The evaluation shows complete utilization of idle nodes by Spark applications.
AB - The scheduling of resources on High-Performance Computing systems (HPC) for compute-intensive scientific applications often results in idle nodes at different points in time due to the difference in node requirements for each application. One option to optimize node usage is to assign all idle nodes to data analytics applications. Scientific applications generate data as output which can be used as input for data analytics applications. So it would be beneficial if both types of applications can be run on the same HPC system. The Invasive Resource Manager (IRM) is an extension of the Simple Linux Utility Resource Manager (SLURM) with dynamic resource management and scheduling capabilities on HPC systems and Apache Spark is an open-source cluster computing framework that is widely used for data analytics applications. This project integrates Apache Spark with the IRM so that data analytics applications can be run on HPC systems with dynamic resource allocation. This work also collects performance data from Spark applications and improves the existing scheduling strategy of the IRM. The integrated system is deployed on SuperMUC, the supercomputer at the Leibniz Supercomputing Centre in Germany for testing and evaluation. This project illustrates the design for integrating data analytics on HPC systems with the additional advantage of improving resource utilization. The evaluation shows complete utilization of idle nodes by Spark applications.
KW - Apache Spark
KW - Data Analytics
KW - Dynamic Resource Allocation
KW - Elastic Scheduling
KW - High Performance Computing
UR - http://www.scopus.com/inward/record.url?scp=85083567628&partnerID=8YFLogxK
U2 - 10.1109/SmartWorld-UIC-ATC-SCALCOM-IOP-SCI.2019.00279
DO - 10.1109/SmartWorld-UIC-ATC-SCALCOM-IOP-SCI.2019.00279
M3 - Conference contribution
AN - SCOPUS:85083567628
T3 - Proceedings - 2019 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Advanced and Trusted Computing, Scalable Computing and Communications, Internet of People and Smart City Innovation, SmartWorld/UIC/ATC/SCALCOM/IOP/SCI 2019
SP - 1553
EP - 1560
BT - Proceedings - 2019 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Advanced and Trusted Computing, Scalable Computing and Communications, Internet of People and Smart City Innovation, SmartWorld/UIC/ATC/SCALCOM/IOP/SCI 2019
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2019 IEEE SmartWorld, Ubiquitous Intelligence and Computing, Advanced and Trusted Computing, Scalable Computing and Communications, Internet of People and Smart City Innovation, SmartWorld/UIC/ATC/SCALCOM/IOP/SCI 2019
Y2 - 19 August 2019 through 23 August 2019
ER -