A workflow runtime environment for manycore parallel architectures

Matthias Janetschek; Radu Prodan; Shajulin Benedict

doi:10.1145/2822332.2822333

A workflow runtime environment for manycore parallel architectures

Matthias Janetschek, Radu Prodan, Shajulin Benedict

University of Innsbruck

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

We introduce a new Manycore Workflow Runtime Environment (MWRE) to efficiently enact traditional scientific workflows on modern manycore computing architectures. In contrast to existing engines that enact workflows acting as external services, MWRE is compiler-based and translates workflows specified in the XMLbased InteroperableWorkflow Intermediate Representation (IWIR) into an equivalent C++-based program. This program efficiently enacts the workflow as a stand-alone executable by means of a new callback mechanism that resolves dependencies, transfers data, and handles composite activities. Experimental results on a number of real-world workflows demonstrate that MWRE clearly outperforms existing Java-based workflow engines designed for distributed (Grid/Cloud) computing infrastructures in terms of enactment time, is generally better than an existing script-based engine for manycore architectures (Swift), and sometimes gets even close to an artificial baseline implementation of the workflows in the standard OpenMP language for shared memory systems.

Original language	English
Title of host publication	Proceedings of WORKS 2015
Subtitle of host publication	10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Association for Computing Machinery, Inc
ISBN (Electronic)	9781450339896
DOIs	https://doi.org/10.1145/2822332.2822333
State	Published - 15 Nov 2015
Externally published	Yes
Event	10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015 - Austin, United States Duration: 15 Nov 2015 → …

Publication series

Name	Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015
Country/Territory	United States
City	Austin
Period	15/11/15 → …

Keywords

Enactment engine
Heterogeneous manycore parallel architectures
Scientific workflows
Source-to-source compiler

Access to Document

10.1145/2822332.2822333

Cite this

Janetschek, M., Prodan, R., & Benedict, S. (2015). A workflow runtime environment for manycore parallel architectures. In Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis Article a1 (Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis). Association for Computing Machinery, Inc. https://doi.org/10.1145/2822332.2822333

Janetschek, Matthias ; Prodan, Radu ; Benedict, Shajulin. / A workflow runtime environment for manycore parallel architectures. Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, Inc, 2015. (Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{4e020812a5184b489449b971f558cad8,

title = "A workflow runtime environment for manycore parallel architectures",

abstract = "We introduce a new Manycore Workflow Runtime Environment (MWRE) to efficiently enact traditional scientific workflows on modern manycore computing architectures. In contrast to existing engines that enact workflows acting as external services, MWRE is compiler-based and translates workflows specified in the XMLbased InteroperableWorkflow Intermediate Representation (IWIR) into an equivalent C++-based program. This program efficiently enacts the workflow as a stand-alone executable by means of a new callback mechanism that resolves dependencies, transfers data, and handles composite activities. Experimental results on a number of real-world workflows demonstrate that MWRE clearly outperforms existing Java-based workflow engines designed for distributed (Grid/Cloud) computing infrastructures in terms of enactment time, is generally better than an existing script-based engine for manycore architectures (Swift), and sometimes gets even close to an artificial baseline implementation of the workflows in the standard OpenMP language for shared memory systems.",

keywords = "Enactment engine, Heterogeneous manycore parallel architectures, Scientific workflows, Source-to-source compiler",

author = "Matthias Janetschek and Radu Prodan and Shajulin Benedict",

note = "Publisher Copyright: {\textcopyright} 2015 ACM.; 10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015 ; Conference date: 15-11-2015",

year = "2015",

month = nov,

day = "15",

doi = "10.1145/2822332.2822333",

language = "English",

series = "Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Association for Computing Machinery, Inc",

booktitle = "Proceedings of WORKS 2015",

}

Janetschek, M, Prodan, R & Benedict, S 2015, A workflow runtime environment for manycore parallel architectures. in Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis., a1, Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis, Association for Computing Machinery, Inc, 10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015, Austin, United States, 15/11/15. https://doi.org/10.1145/2822332.2822333

A workflow runtime environment for manycore parallel architectures. / Janetschek, Matthias; Prodan, Radu; Benedict, Shajulin.
Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, Inc, 2015. a1 (Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A workflow runtime environment for manycore parallel architectures

AU - Janetschek, Matthias

AU - Prodan, Radu

AU - Benedict, Shajulin

PY - 2015/11/15

Y1 - 2015/11/15

N2 - We introduce a new Manycore Workflow Runtime Environment (MWRE) to efficiently enact traditional scientific workflows on modern manycore computing architectures. In contrast to existing engines that enact workflows acting as external services, MWRE is compiler-based and translates workflows specified in the XMLbased InteroperableWorkflow Intermediate Representation (IWIR) into an equivalent C++-based program. This program efficiently enacts the workflow as a stand-alone executable by means of a new callback mechanism that resolves dependencies, transfers data, and handles composite activities. Experimental results on a number of real-world workflows demonstrate that MWRE clearly outperforms existing Java-based workflow engines designed for distributed (Grid/Cloud) computing infrastructures in terms of enactment time, is generally better than an existing script-based engine for manycore architectures (Swift), and sometimes gets even close to an artificial baseline implementation of the workflows in the standard OpenMP language for shared memory systems.

AB - We introduce a new Manycore Workflow Runtime Environment (MWRE) to efficiently enact traditional scientific workflows on modern manycore computing architectures. In contrast to existing engines that enact workflows acting as external services, MWRE is compiler-based and translates workflows specified in the XMLbased InteroperableWorkflow Intermediate Representation (IWIR) into an equivalent C++-based program. This program efficiently enacts the workflow as a stand-alone executable by means of a new callback mechanism that resolves dependencies, transfers data, and handles composite activities. Experimental results on a number of real-world workflows demonstrate that MWRE clearly outperforms existing Java-based workflow engines designed for distributed (Grid/Cloud) computing infrastructures in terms of enactment time, is generally better than an existing script-based engine for manycore architectures (Swift), and sometimes gets even close to an artificial baseline implementation of the workflows in the standard OpenMP language for shared memory systems.

KW - Enactment engine

KW - Heterogeneous manycore parallel architectures

KW - Scientific workflows

KW - Source-to-source compiler

UR - http://www.scopus.com/inward/record.url?scp=84960855326&partnerID=8YFLogxK

U2 - 10.1145/2822332.2822333

DO - 10.1145/2822332.2822333

M3 - Conference contribution

AN - SCOPUS:84960855326

T3 - Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis

BT - Proceedings of WORKS 2015

PB - Association for Computing Machinery, Inc

T2 - 10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015

Y2 - 15 November 2015

ER -

Janetschek M, Prodan R, Benedict S. A workflow runtime environment for manycore parallel architectures. In Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, Inc. 2015. a1. (Proceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1145/2822332.2822333

A workflow runtime environment for manycore parallel architectures

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this