BugsJS: A benchmark of javascript bugs

Peter Gyimesi; Bela Vancsics; Andrea Stocco; Davood Mazinanian; Arpad Beszedes; Rudolf Ferenc; Ali Mesbah

doi:10.1109/ICST.2019.00019

BugsJS: A benchmark of javascript bugs

Peter Gyimesi, Bela Vancsics, Andrea Stocco, Davood Mazinanian, Arpad Beszedes, Rudolf Ferenc, Ali Mesbah

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

80 Scopus citations

Abstract

JavaScript is a popular programming language that is also error-prone due to its asynchronous, dynamic, and loosely-typed nature. In recent years, numerous techniques have been proposed for analyzing and testing JavaScript applications. However, our survey of the literature in this area revealed that the proposed techniques are often evaluated on different datasets of programs and bugs. The lack of a commonly used benchmark limits the ability to perform fair and unbiased comparisons for assessing the efficacy of new techniques. To fill this gap, we propose BugsJS, a benchmark of 453 real, manually validated JavaScript bugs from 10 popular JavaScript server-side programs, comprising 444k LOC in total. Each bug is accompanied by its bug report, the test cases that detect it, as well as the patch that fixes it. BugsJS features a rich interface for accessing the faulty and fixed versions of the programs and executing the corresponding test cases, which facilitates conducting highly-reproducible empirical studies and comparisons of JavaScript analysis and testing tools.

Original language	English
Title of host publication	Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	90-101
Number of pages	12
ISBN (Electronic)	9781728117355
DOIs	https://doi.org/10.1109/ICST.2019.00019
State	Published - Apr 2019
Externally published	Yes
Event	12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019 - Xi'an, China Duration: 22 Apr 2019 → 27 Apr 2019

Publication series

Name	Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

Conference

Conference	12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019
Country/Territory	China
City	Xi'an
Period	22/04/19 → 27/04/19

Keywords

Benchmark
Bug database
BugsJS
JavaScript
Literature survey
Real bugs
Reproducibility

Access to Document

10.1109/ICST.2019.00019

Cite this

Gyimesi, P., Vancsics, B., Stocco, A., Mazinanian, D., Beszedes, A., Ferenc, R., & Mesbah, A. (2019). BugsJS: A benchmark of javascript bugs. In Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019 (pp. 90-101). Article 8730197 (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICST.2019.00019

Gyimesi, Peter ; Vancsics, Bela ; Stocco, Andrea et al. / BugsJS : A benchmark of javascript bugs. Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 90-101 (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019).

@inproceedings{9b36b62b1e5e47dd879c972f2c030f13,

title = "BugsJS: A benchmark of javascript bugs",

abstract = "JavaScript is a popular programming language that is also error-prone due to its asynchronous, dynamic, and loosely-typed nature. In recent years, numerous techniques have been proposed for analyzing and testing JavaScript applications. However, our survey of the literature in this area revealed that the proposed techniques are often evaluated on different datasets of programs and bugs. The lack of a commonly used benchmark limits the ability to perform fair and unbiased comparisons for assessing the efficacy of new techniques. To fill this gap, we propose BugsJS, a benchmark of 453 real, manually validated JavaScript bugs from 10 popular JavaScript server-side programs, comprising 444k LOC in total. Each bug is accompanied by its bug report, the test cases that detect it, as well as the patch that fixes it. BugsJS features a rich interface for accessing the faulty and fixed versions of the programs and executing the corresponding test cases, which facilitates conducting highly-reproducible empirical studies and comparisons of JavaScript analysis and testing tools.",

keywords = "Benchmark, Bug database, BugsJS, JavaScript, Literature survey, Real bugs, Reproducibility",

author = "Peter Gyimesi and Bela Vancsics and Andrea Stocco and Davood Mazinanian and Arpad Beszedes and Rudolf Ferenc and Ali Mesbah",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019 ; Conference date: 22-04-2019 Through 27-04-2019",

year = "2019",

month = apr,

doi = "10.1109/ICST.2019.00019",

language = "English",

series = "Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "90--101",

booktitle = "Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019",

}

Gyimesi, P, Vancsics, B, Stocco, A, Mazinanian, D, Beszedes, A, Ferenc, R & Mesbah, A 2019, BugsJS: A benchmark of javascript bugs. in Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019., 8730197, Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019, Institute of Electrical and Electronics Engineers Inc., pp. 90-101, 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019, Xi'an, China, 22/04/19. https://doi.org/10.1109/ICST.2019.00019

BugsJS: A benchmark of javascript bugs. / Gyimesi, Peter; Vancsics, Bela; Stocco, Andrea et al.
Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 90-101 8730197 (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - BugsJS

T2 - 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019

AU - Gyimesi, Peter

AU - Vancsics, Bela

AU - Stocco, Andrea

AU - Mazinanian, Davood

AU - Beszedes, Arpad

AU - Ferenc, Rudolf

AU - Mesbah, Ali

PY - 2019/4

Y1 - 2019/4

N2 - JavaScript is a popular programming language that is also error-prone due to its asynchronous, dynamic, and loosely-typed nature. In recent years, numerous techniques have been proposed for analyzing and testing JavaScript applications. However, our survey of the literature in this area revealed that the proposed techniques are often evaluated on different datasets of programs and bugs. The lack of a commonly used benchmark limits the ability to perform fair and unbiased comparisons for assessing the efficacy of new techniques. To fill this gap, we propose BugsJS, a benchmark of 453 real, manually validated JavaScript bugs from 10 popular JavaScript server-side programs, comprising 444k LOC in total. Each bug is accompanied by its bug report, the test cases that detect it, as well as the patch that fixes it. BugsJS features a rich interface for accessing the faulty and fixed versions of the programs and executing the corresponding test cases, which facilitates conducting highly-reproducible empirical studies and comparisons of JavaScript analysis and testing tools.

AB - JavaScript is a popular programming language that is also error-prone due to its asynchronous, dynamic, and loosely-typed nature. In recent years, numerous techniques have been proposed for analyzing and testing JavaScript applications. However, our survey of the literature in this area revealed that the proposed techniques are often evaluated on different datasets of programs and bugs. The lack of a commonly used benchmark limits the ability to perform fair and unbiased comparisons for assessing the efficacy of new techniques. To fill this gap, we propose BugsJS, a benchmark of 453 real, manually validated JavaScript bugs from 10 popular JavaScript server-side programs, comprising 444k LOC in total. Each bug is accompanied by its bug report, the test cases that detect it, as well as the patch that fixes it. BugsJS features a rich interface for accessing the faulty and fixed versions of the programs and executing the corresponding test cases, which facilitates conducting highly-reproducible empirical studies and comparisons of JavaScript analysis and testing tools.

KW - Benchmark

KW - Bug database

KW - BugsJS

KW - JavaScript

KW - Literature survey

KW - Real bugs

KW - Reproducibility

UR - http://www.scopus.com/inward/record.url?scp=85067491657&partnerID=8YFLogxK

U2 - 10.1109/ICST.2019.00019

DO - 10.1109/ICST.2019.00019

M3 - Conference contribution

AN - SCOPUS:85067491657

T3 - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

SP - 90

EP - 101

BT - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 22 April 2019 through 27 April 2019

ER -

Gyimesi P, Vancsics B, Stocco A, Mazinanian D, Beszedes A, Ferenc R et al. BugsJS: A benchmark of javascript bugs. In Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 90-101. 8730197. (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019). doi: 10.1109/ICST.2019.00019

BugsJS: A benchmark of javascript bugs

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this