A workflow runtime environment for manycore parallel architectures

M. Janetschek, R. Prodan, S. Benedict

Research output: Contribution to journalArticlepeer-review

10 Scopus citations

Abstract

We introduce a new Manycore Workflow Runtime Environment (MWRE) to efficiently enact traditional scientific workflows on modern manycore computing architectures. MWRE is compiler-based and translates workflows specified in the XML-based Interoperable Workflow Intermediate Representation (IWIR) into an equivalent C++-based program. This program efficiently enacts the workflow as a stand-alone executable by means of a new callback mechanism that resolves dependencies, transfers data, and handles composite activities. Furthermore, a core feature of MWRE is explicit support for full-ahead scheduling and enactment. Experimental results on a number of real-world workflows demonstrate that MWRE clearly outperforms existing Java-based workflow engines designed for distributed (Grid or Cloud) computing infrastructures in terms of enactment time, is generally better than an existing script-based engine for manycore architectures (Swift), and sometimes gets even close to an artificial baseline implementation of the workflows in the standard OpenMP language for shared memory systems. Experimental results also show that full-ahead scheduling with MWRE using a state-of-the-art heuristic can improve the workflow performance up to 40%.

Original languageEnglish
Pages (from-to)330-347
Number of pages18
JournalFuture Generation Computer Systems
Volume75
DOIs
StatePublished - Oct 2017
Externally publishedYes

Keywords

  • Full-ahead scheduling
  • Manycores
  • Scientific workflows
  • Workflow execution plan

Fingerprint

Dive into the research topics of 'A workflow runtime environment for manycore parallel architectures'. Together they form a unique fingerprint.

Cite this