TY - GEN
T1 - FuzzTastic
T2 - 44th ACM/IEEE International Conference on Software Engineering: Companion, ICSE-Companion 2022
AU - Lipp, Stephan
AU - Elsner, Daniel
AU - Hutzelmann, Thomas
AU - Banescu, Sebastian
AU - Pretschner, Alexander
AU - Bohme, Marcel
N1 - Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - Performing sound and fair fuzzer evaluations can be challenging, not only because of the randomness involved in fuzzing, but also due to the large number of fuzz tests generated. Existing evaluations use code coverage as a proxy measure for fuzzing effectiveness. Yet, instead of considering coverage of all generated fuzz inputs, they only consider the inputs stored in the fuzzer queue. However, as we show in this paper, this approach can lead to biased assessments due to path collisions. Therefore, we developed FuzzTastic, a fuzzer-agnostic coverage analyzer that allows practitioners and researchers to perform uniform fuzzer evaluations that are not affected by such collisions. In addition, its time-stamped coverage-probing approach enables frequency-based coverage analysis to identify barely tested source code and to visualize fuzzing progress over time and across code. To foster further studies in this field, we make FuzzTastic, together with a benchmark dataset worth ~12 CPU-years of fuzzing, publicly available; the demo video can be found at https://youtu.be/Lm-eBx0aePA.
AB - Performing sound and fair fuzzer evaluations can be challenging, not only because of the randomness involved in fuzzing, but also due to the large number of fuzz tests generated. Existing evaluations use code coverage as a proxy measure for fuzzing effectiveness. Yet, instead of considering coverage of all generated fuzz inputs, they only consider the inputs stored in the fuzzer queue. However, as we show in this paper, this approach can lead to biased assessments due to path collisions. Therefore, we developed FuzzTastic, a fuzzer-agnostic coverage analyzer that allows practitioners and researchers to perform uniform fuzzer evaluations that are not affected by such collisions. In addition, its time-stamped coverage-probing approach enables frequency-based coverage analysis to identify barely tested source code and to visualize fuzzing progress over time and across code. To foster further studies in this field, we make FuzzTastic, together with a benchmark dataset worth ~12 CPU-years of fuzzing, publicly available; the demo video can be found at https://youtu.be/Lm-eBx0aePA.
KW - • Security and privacy → Software security engineering.
UR - http://www.scopus.com/inward/record.url?scp=85132357532&partnerID=8YFLogxK
U2 - 10.1109/ICSE-Companion55297.2022.9793832
DO - 10.1109/ICSE-Companion55297.2022.9793832
M3 - Conference contribution
AN - SCOPUS:85132357532
T3 - Proceedings - International Conference on Software Engineering
SP - 75
EP - 79
BT - Proceedings - 2022 ACM/IEEE 44th International Conference on Software Engineering
PB - IEEE Computer Society
Y2 - 22 May 2022 through 27 May 2022
ER -