TY - GEN
T1 - Challenger 2.0
T2 - 18th ACM International Conference on Distributed and Event-Based Systems, DEBS 2024
AU - Tahir, Jawad
AU - Baili, Chiheb
AU - Svaral, Matej
AU - Friedlein, Johannes
AU - Doblander, Christoph
AU - Jacobsen, Hans Arno
N1 - Publisher Copyright:
© 2024 Owner/Author.
PY - 2024/6/24
Y1 - 2024/6/24
N2 - The DEBS Grand Challenge (GC) is a yearly programming competition organized by the DEBS community. The participants of the GC are provided with a dataset and are required to build a solution generating insights from the data. Participants deploy their solutions on the provided virtual machines (VMs). The dataset is disseminated and the solutions' performance is measured using Challenger, an RPC-based service. Developer surveys show a lower adaption of RPC, which may limit the audience of the GC. Furthermore, provisioning of VMs blocks the compute resources, setting a limit on the number of participants. Lastly, Challenger lacks the functionality to test the fault-tolerance capabilities of the solutions, which is a strict non-functional requirement for the solutions. In this paper, we propose, implement, and demonstrate changes in the current implementation of Challenger that address the aforementioned issues and introduce Challenger 2.0, which is a one-stop solution for data dissemination, performance benchmarking, and automated deployments in the GC. For Challenger 2.0, we port its API to the REST framework from RPC to broaden the target audience of the challenge. Furthermore, this change removes the theoretical limit on the number of participants partaking in the GC, thanks to the container-based deployment methodology. Lastly, Challenger 2.0 injects faults during evaluations to verify the fault tolerance of the solutions.
AB - The DEBS Grand Challenge (GC) is a yearly programming competition organized by the DEBS community. The participants of the GC are provided with a dataset and are required to build a solution generating insights from the data. Participants deploy their solutions on the provided virtual machines (VMs). The dataset is disseminated and the solutions' performance is measured using Challenger, an RPC-based service. Developer surveys show a lower adaption of RPC, which may limit the audience of the GC. Furthermore, provisioning of VMs blocks the compute resources, setting a limit on the number of participants. Lastly, Challenger lacks the functionality to test the fault-tolerance capabilities of the solutions, which is a strict non-functional requirement for the solutions. In this paper, we propose, implement, and demonstrate changes in the current implementation of Challenger that address the aforementioned issues and introduce Challenger 2.0, which is a one-stop solution for data dissemination, performance benchmarking, and automated deployments in the GC. For Challenger 2.0, we port its API to the REST framework from RPC to broaden the target audience of the challenge. Furthermore, this change removes the theoretical limit on the number of participants partaking in the GC, thanks to the container-based deployment methodology. Lastly, Challenger 2.0 injects faults during evaluations to verify the fault tolerance of the solutions.
KW - Benchmarking
KW - DEBS Grand Challenge
KW - Fault-tolerance
UR - http://www.scopus.com/inward/record.url?scp=85200678615&partnerID=8YFLogxK
U2 - 10.1145/3629104.3666027
DO - 10.1145/3629104.3666027
M3 - Conference contribution
AN - SCOPUS:85200678615
T3 - DEBS 2024 - Proceedings of the 18th ACM International Conference on Distributed and Event-Based Systems
SP - 6
EP - 17
BT - DEBS 2024 - Proceedings of the 18th ACM International Conference on Distributed and Event-Based Systems
PB - Association for Computing Machinery, Inc
Y2 - 25 June 2024 through 28 June 2024
ER -