TY - GEN
T1 - Fex
T2 - 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2017
AU - Oleksenko, Oleksii
AU - Kuvaiskii, Dmitrii
AU - Bhatotia, Pramod
AU - Fetzer, Christof
N1 - Publisher Copyright:
© 2017 IEEE.
PY - 2017/8/30
Y1 - 2017/8/30
N2 - Software systems research relies on experimental evaluation to assess the effectiveness of newly developed solutions. However, the existing evaluation frameworks are rigid (do not allow creation of new experiments), often simplistic (may not reveal issues that appear in real-world applications), and can be inconsistent (do not guarantee reproducibility of experiments across platforms). This paper presents Fex, a software systems evaluation framework that addresses these limitations. Fex is extensible (can be easily extended with custom experiment types), practical (supports composition of different benchmark suites and real-world applications), and reproducible (it is built on container technology to guarantee the same software stack across platforms). We show that Fex achieves these design goals with minimal end-user effort-for instance, adding Nginx web-server to evaluation requires only 160 LoC. Going forward, we discuss the architecture of the framework, explain its interface, show common usage scenarios, and evaluate the efforts for writing various custom extensions.
AB - Software systems research relies on experimental evaluation to assess the effectiveness of newly developed solutions. However, the existing evaluation frameworks are rigid (do not allow creation of new experiments), often simplistic (may not reveal issues that appear in real-world applications), and can be inconsistent (do not guarantee reproducibility of experiments across platforms). This paper presents Fex, a software systems evaluation framework that addresses these limitations. Fex is extensible (can be easily extended with custom experiment types), practical (supports composition of different benchmark suites and real-world applications), and reproducible (it is built on container technology to guarantee the same software stack across platforms). We show that Fex achieves these design goals with minimal end-user effort-for instance, adding Nginx web-server to evaluation requires only 160 LoC. Going forward, we discuss the architecture of the framework, explain its interface, show common usage scenarios, and evaluate the efforts for writing various custom extensions.
KW - Benchmarks
KW - experiment automation
KW - performance
UR - http://www.scopus.com/inward/record.url?scp=85019208293&partnerID=8YFLogxK
U2 - 10.1109/DSN.2017.25
DO - 10.1109/DSN.2017.25
M3 - Conference contribution
AN - SCOPUS:85019208293
T3 - Proceedings - 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2017
SP - 543
EP - 550
BT - Proceedings - 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2017
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 26 June 2017 through 29 June 2017
ER -