Interpolation can hurt robust generalization even when there is no noise

Konstantin Donhauser; Alexandru Tifrea; Michael Aerni; Reinhard Heckel; Fanny Yang

Interpolation can hurt robust generalization even when there is no noise

Konstantin Donhauser, Alexandru Tifrea, Michael Aerni, Reinhard Heckel, Fanny Yang

Associate Professorship of Machine Learning

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

9 Scopus citations

Abstract

Numerous recent works show that overparameterization implicitly reduces variance for min-norm interpolators and max-margin classifiers. These findings suggest that ridge regularization has vanishing benefits in high dimensions. We challenge this narrative by showing that, even in the absence of noise, avoiding interpolation through ridge regularization can significantly improve generalization. We prove this phenomenon for the robust risk of both linear regression and classification, and hence provide the first theoretical result on robust overfitting.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Editors	Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Publisher	Neural information processing systems foundation
Pages	23465-23477
Number of pages	13
ISBN (Electronic)	9781713845393
State	Published - 2021
Event	35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online Duration: 6 Dec 2021 → 14 Dec 2021

Publication series

Name	Advances in Neural Information Processing Systems
Volume	28
ISSN (Print)	1049-5258

Conference

Conference	35th Conference on Neural Information Processing Systems, NeurIPS 2021
City	Virtual, Online
Period	6/12/21 → 14/12/21

Cite this

Donhauser, K., Tifrea, A., Aerni, M., Heckel, R., & Yang, F. (2021). Interpolation can hurt robust generalization even when there is no noise. In MA. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021 (pp. 23465-23477). (Advances in Neural Information Processing Systems; Vol. 28). Neural information processing systems foundation.

Donhauser, Konstantin ; Tifrea, Alexandru ; Aerni, Michael et al. / Interpolation can hurt robust generalization even when there is no noise. Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. editor / Marc'Aurelio Ranzato ; Alina Beygelzimer ; Yann Dauphin ; Percy S. Liang ; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. pp. 23465-23477 (Advances in Neural Information Processing Systems).

@inproceedings{a394003e884a4336ab709e0b9929a8af,

title = "Interpolation can hurt robust generalization even when there is no noise",

abstract = "Numerous recent works show that overparameterization implicitly reduces variance for min-norm interpolators and max-margin classifiers. These findings suggest that ridge regularization has vanishing benefits in high dimensions. We challenge this narrative by showing that, even in the absence of noise, avoiding interpolation through ridge regularization can significantly improve generalization. We prove this phenomenon for the robust risk of both linear regression and classification, and hence provide the first theoretical result on robust overfitting.",

author = "Konstantin Donhauser and Alexandru Tifrea and Michael Aerni and Reinhard Heckel and Fanny Yang",

note = "Publisher Copyright: {\textcopyright} 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

year = "2021",

language = "English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

pages = "23465--23477",

editor = "Marc'Aurelio Ranzato and Alina Beygelzimer and Yann Dauphin and Liang, {Percy S.} and {Wortman Vaughan}, Jenn",

booktitle = "Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021",

}

Donhauser, K, Tifrea, A, Aerni, M, Heckel, R & Yang, F 2021, Interpolation can hurt robust generalization even when there is no noise. in MA Ranzato, A Beygelzimer, Y Dauphin, PS Liang & J Wortman Vaughan (eds), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Advances in Neural Information Processing Systems, vol. 28, Neural information processing systems foundation, pp. 23465-23477, 35th Conference on Neural Information Processing Systems, NeurIPS 2021, Virtual, Online, 6/12/21.

Interpolation can hurt robust generalization even when there is no noise. / Donhauser, Konstantin; Tifrea, Alexandru; Aerni, Michael et al.
Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. ed. / Marc'Aurelio Ranzato; Alina Beygelzimer; Yann Dauphin; Percy S. Liang; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. p. 23465-23477 (Advances in Neural Information Processing Systems; Vol. 28).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Interpolation can hurt robust generalization even when there is no noise

AU - Donhauser, Konstantin

AU - Tifrea, Alexandru

AU - Aerni, Michael

AU - Heckel, Reinhard

AU - Yang, Fanny

PY - 2021

Y1 - 2021

N2 - Numerous recent works show that overparameterization implicitly reduces variance for min-norm interpolators and max-margin classifiers. These findings suggest that ridge regularization has vanishing benefits in high dimensions. We challenge this narrative by showing that, even in the absence of noise, avoiding interpolation through ridge regularization can significantly improve generalization. We prove this phenomenon for the robust risk of both linear regression and classification, and hence provide the first theoretical result on robust overfitting.

AB - Numerous recent works show that overparameterization implicitly reduces variance for min-norm interpolators and max-margin classifiers. These findings suggest that ridge regularization has vanishing benefits in high dimensions. We challenge this narrative by showing that, even in the absence of noise, avoiding interpolation through ridge regularization can significantly improve generalization. We prove this phenomenon for the robust risk of both linear regression and classification, and hence provide the first theoretical result on robust overfitting.

UR - http://www.scopus.com/inward/record.url?scp=85132038260&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85132038260

T3 - Advances in Neural Information Processing Systems

SP - 23465

EP - 23477

BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

A2 - Ranzato, Marc'Aurelio

A2 - Beygelzimer, Alina

A2 - Dauphin, Yann

A2 - Liang, Percy S.

A2 - Wortman Vaughan, Jenn

PB - Neural information processing systems foundation

T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

Y2 - 6 December 2021 through 14 December 2021

ER -

Donhauser K, Tifrea A, Aerni M, Heckel R, Yang F. Interpolation can hurt robust generalization even when there is no noise. In Ranzato MA, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, editors, Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Neural information processing systems foundation. 2021. p. 23465-23477. (Advances in Neural Information Processing Systems).

Interpolation can hurt robust generalization even when there is no noise

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this