Energy Efficient Frequency Scaling on GPUs in Heterogeneous HPC Systems

With most major corporations and research institutions having pledged to support sustainability goals for High Performance Computing (HPC), energy efficiency is a critical factor when evaluating heterogeneous HPC systems. However, many popular hardware performance & energy measurement frameworks, such as LIKWID, and benchmarks, such as the STREAM or the hipBone benchmark, do not or not fully support execution on heterogeneous systems containing AMD or NVIDIA Graphical Processing Units (GPUs), leading to a gap with regards to the understanding the relationship between frequency, performance and energy. We aim at closing this gap by extending the performance measurement framework LIKWID to support both AMD and NVIDIA GPUs. We run the STREAM and hipBone benchmark on AMD and NVIDIA GPUs at different GPU core frequencies. We show that the minimum period between two measurements for our GPU is at least 100ms and that GPUs have a sweet spot with regards to energy consumption at approximately 75% of their maximum frequency with energy savings up to 30% at a performance overhead between 0.72% and 3.12%.