Are Transformers More Robust? Towards Exact Robustness Verification for Transformers

Brian Hsuan Cheng Liao; Chih Hong Cheng; Hasan Esen; Alois Knoll

doi:10.1007/978-3-031-40923-3_8

Are Transformers More Robust? Towards Exact Robustness Verification for Transformers

Brian Hsuan Cheng Liao, Chih Hong Cheng, Hasan Esen, Alois Knoll

nformatics 6 - Chair of IRobotics, Artificial Intelligence and Real-time Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

As an emerging type of Neural Networks (NNs), Transformers are used in many domains ranging from Natural Language Processing to Autonomous Driving. In this paper, we study the robustness problem of Transformers, a key characteristic as low robustness may cause safety concerns. Specifically, we focus on Sparsemax-based Transformers and reduce the finding of their maximum robustness to a Mixed Integer Quadratically Constrained Programming (MIQCP) problem. We also design two pre-processing heuristics that can be embedded in the MIQCP encoding and substantially accelerate its solving. We then conduct experiments using the application of Land Departure Warning to compare the robustness of Sparsemax-based Transformers against that of the more conventional Multi-Layer-Perceptron (MLP) NNs. To our surprise, Transformers are not necessarily more robust, leading to profound considerations in selecting appropriate NN architectures for safety-critical domain applications.

Original language	English
Title of host publication	Computer Safety, Reliability, and Security - 42nd International Conference, SAFECOMP 2023, Proceedings
Editors	Jérémie Guiochet, Stefano Tonetta, Friedemann Bitsch
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	89-103
Number of pages	15
ISBN (Print)	9783031409226
DOIs	https://doi.org/10.1007/978-3-031-40923-3_8
State	Published - 2023
Event	Proceedings of the 42nd International Conference on Computer Safety, Reliability and Security, SAFECOMP 2023 - Toulouse, France Duration: 20 Sep 2023 → 22 Sep 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14181 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Proceedings of the 42nd International Conference on Computer Safety, Reliability and Security, SAFECOMP 2023
Country/Territory	France
City	Toulouse
Period	20/09/23 → 22/09/23

Keywords

Autonomous Driving
Lane Departure Warning
NN verification
Robustness
Transformers

Access to Document

10.1007/978-3-031-40923-3_8

Cite this

Liao, B. H. C., Cheng, C. H., Esen, H., & Knoll, A. (2023). Are Transformers More Robust? Towards Exact Robustness Verification for Transformers. In J. Guiochet, S. Tonetta, & F. Bitsch (Eds.), Computer Safety, Reliability, and Security - 42nd International Conference, SAFECOMP 2023, Proceedings (pp. 89-103). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14181 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-40923-3_8

Liao, Brian Hsuan Cheng ; Cheng, Chih Hong ; Esen, Hasan et al. / Are Transformers More Robust? Towards Exact Robustness Verification for Transformers. Computer Safety, Reliability, and Security - 42nd International Conference, SAFECOMP 2023, Proceedings. editor / Jérémie Guiochet ; Stefano Tonetta ; Friedemann Bitsch. Springer Science and Business Media Deutschland GmbH, 2023. pp. 89-103 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{d9baa497eabb4cc2ad453c9338743766,

title = "Are Transformers More Robust? Towards Exact Robustness Verification for Transformers",

abstract = "As an emerging type of Neural Networks (NNs), Transformers are used in many domains ranging from Natural Language Processing to Autonomous Driving. In this paper, we study the robustness problem of Transformers, a key characteristic as low robustness may cause safety concerns. Specifically, we focus on Sparsemax-based Transformers and reduce the finding of their maximum robustness to a Mixed Integer Quadratically Constrained Programming (MIQCP) problem. We also design two pre-processing heuristics that can be embedded in the MIQCP encoding and substantially accelerate its solving. We then conduct experiments using the application of Land Departure Warning to compare the robustness of Sparsemax-based Transformers against that of the more conventional Multi-Layer-Perceptron (MLP) NNs. To our surprise, Transformers are not necessarily more robust, leading to profound considerations in selecting appropriate NN architectures for safety-critical domain applications.",

keywords = "Autonomous Driving, Lane Departure Warning, NN verification, Robustness, Transformers",

author = "Liao, {Brian Hsuan Cheng} and Cheng, {Chih Hong} and Hasan Esen and Alois Knoll",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; Proceedings of the 42nd International Conference on Computer Safety, Reliability and Security, SAFECOMP 2023 ; Conference date: 20-09-2023 Through 22-09-2023",

year = "2023",

doi = "10.1007/978-3-031-40923-3_8",

language = "English",

isbn = "9783031409226",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "89--103",

editor = "J{\'e}r{\'e}mie Guiochet and Stefano Tonetta and Friedemann Bitsch",

booktitle = "Computer Safety, Reliability, and Security - 42nd International Conference, SAFECOMP 2023, Proceedings",

}

Liao, BHC, Cheng, CH, Esen, H & Knoll, A 2023, Are Transformers More Robust? Towards Exact Robustness Verification for Transformers. in J Guiochet, S Tonetta & F Bitsch (eds), Computer Safety, Reliability, and Security - 42nd International Conference, SAFECOMP 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14181 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 89-103, Proceedings of the 42nd International Conference on Computer Safety, Reliability and Security, SAFECOMP 2023, Toulouse, France, 20/09/23. https://doi.org/10.1007/978-3-031-40923-3_8

Are Transformers More Robust? Towards Exact Robustness Verification for Transformers. / Liao, Brian Hsuan Cheng; Cheng, Chih Hong; Esen, Hasan et al.
Computer Safety, Reliability, and Security - 42nd International Conference, SAFECOMP 2023, Proceedings. ed. / Jérémie Guiochet; Stefano Tonetta; Friedemann Bitsch. Springer Science and Business Media Deutschland GmbH, 2023. p. 89-103 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14181 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Are Transformers More Robust? Towards Exact Robustness Verification for Transformers

AU - Liao, Brian Hsuan Cheng

AU - Cheng, Chih Hong

AU - Esen, Hasan

AU - Knoll, Alois

PY - 2023

Y1 - 2023

N2 - As an emerging type of Neural Networks (NNs), Transformers are used in many domains ranging from Natural Language Processing to Autonomous Driving. In this paper, we study the robustness problem of Transformers, a key characteristic as low robustness may cause safety concerns. Specifically, we focus on Sparsemax-based Transformers and reduce the finding of their maximum robustness to a Mixed Integer Quadratically Constrained Programming (MIQCP) problem. We also design two pre-processing heuristics that can be embedded in the MIQCP encoding and substantially accelerate its solving. We then conduct experiments using the application of Land Departure Warning to compare the robustness of Sparsemax-based Transformers against that of the more conventional Multi-Layer-Perceptron (MLP) NNs. To our surprise, Transformers are not necessarily more robust, leading to profound considerations in selecting appropriate NN architectures for safety-critical domain applications.

AB - As an emerging type of Neural Networks (NNs), Transformers are used in many domains ranging from Natural Language Processing to Autonomous Driving. In this paper, we study the robustness problem of Transformers, a key characteristic as low robustness may cause safety concerns. Specifically, we focus on Sparsemax-based Transformers and reduce the finding of their maximum robustness to a Mixed Integer Quadratically Constrained Programming (MIQCP) problem. We also design two pre-processing heuristics that can be embedded in the MIQCP encoding and substantially accelerate its solving. We then conduct experiments using the application of Land Departure Warning to compare the robustness of Sparsemax-based Transformers against that of the more conventional Multi-Layer-Perceptron (MLP) NNs. To our surprise, Transformers are not necessarily more robust, leading to profound considerations in selecting appropriate NN architectures for safety-critical domain applications.

KW - Autonomous Driving

KW - Lane Departure Warning

KW - NN verification

KW - Robustness

KW - Transformers

UR - http://www.scopus.com/inward/record.url?scp=85172138823&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-40923-3_8

DO - 10.1007/978-3-031-40923-3_8

M3 - Conference contribution

AN - SCOPUS:85172138823

SN - 9783031409226

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 89

EP - 103

BT - Computer Safety, Reliability, and Security - 42nd International Conference, SAFECOMP 2023, Proceedings

A2 - Guiochet, Jérémie

A2 - Tonetta, Stefano

A2 - Bitsch, Friedemann

PB - Springer Science and Business Media Deutschland GmbH

T2 - Proceedings of the 42nd International Conference on Computer Safety, Reliability and Security, SAFECOMP 2023

Y2 - 20 September 2023 through 22 September 2023

ER -

Liao BHC, Cheng CH, Esen H, Knoll A. Are Transformers More Robust? Towards Exact Robustness Verification for Transformers. In Guiochet J, Tonetta S, Bitsch F, editors, Computer Safety, Reliability, and Security - 42nd International Conference, SAFECOMP 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 89-103. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-40923-3_8