An Emulation Layer for Dynamic Resources with MPI Sessions

Jan Fecht, Martin Schreiber, Martin Schulz, Howard Pritchard, Daniel J. Holmes

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations

Abstract

The current static job scheduling on supercomputers for MPI-based applications is well known to be a limiting factor for the exploitation of a system’s top performance in terms of application throughput. Hence, allowing fully flexible and dynamically varying job sizes would provide multiple advantages compared to the current approach, e.g., by prioritizing jobs dynamically and optimizing resource usage by transferring resources economically. A critical step in achieving dynamic resource management with MPI on supercomputers is the development of sound and robust interfaces between MPI applications and the runtime system. Our approach extends the concept of MPI Sessions, a new concept introduced with MPI 4.0, by adding new features to support varying computing resources via the MPI process set abstraction. We then show how these features can be used, as a proof of concept, to request (active) and cope with (passive) varying resources from an application’s perspective. To validate of our approach, we develop libmpidynres, a C library providing an emulated MPI Sessions environment on top of existing MPI implementations without MPI Sessions support, which we then use to integrate our proposed extensions to the interface specification. Using this proof-of-concept environment, we show how an MPI Sessions enabled application can use process sets to handle dynamically varying resources.

Original languageEnglish
Title of host publicationHigh Performance Computing. ISC High Performance 2022 International Workshops - Revised Selected Papers
EditorsHartwig Anzt, Amanda Bienz, Piotr Luszczek, Marc Baboulin
PublisherSpringer Science and Business Media Deutschland GmbH
Pages147-161
Number of pages15
ISBN (Print)9783031232190
DOIs
StatePublished - 2022
Event37th International Conference on High Performance Computing , ISC High Performance 2022 - Hamburg, Germany
Duration: 29 May 20222 Jun 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13387 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference37th International Conference on High Performance Computing , ISC High Performance 2022
Country/TerritoryGermany
CityHamburg
Period29/05/222/06/22

Keywords

  • Dynamic resources
  • MPI
  • MPI Sessions
  • Resource management

Fingerprint

Dive into the research topics of 'An Emulation Layer for Dynamic Resources with MPI Sessions'. Together they form a unique fingerprint.

Cite this