Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks

Matthias Kissel; Klaus Diepold

doi:10.1007/978-3-030-46147-8_24

Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks

Matthias Kissel, Klaus Diepold

Chair of Data Processing

Technical University of Munich

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

With Sobolev Training, neural networks are trained to fit target output values as well as target derivatives with respect to the inputs. This leads to better generalization and fewer required training examples for certain problems. In this paper, we present a training pipeline that enables Sobolev Training for regression problems where target derivatives are not directly available. Thus, we propose to use a least-squares estimate of the target derivatives based on function values of neighboring training samples. We show for a variety of black-box function regression tasks that our training pipeline achieves smaller test errors compared to the traditional training method. Since our method has no additional requirements on the data collection process, it has great potential to improve the results for various regression tasks.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Proceedings
Editors	Ulf Brefeld, Elisa Fromont, Andreas Hotho, Arno Knobbe, Marloes Maathuis, Céline Robardet
Publisher	Springer
Pages	399-414
Number of pages	16
ISBN (Print)	9783030461461
DOIs	https://doi.org/10.1007/978-3-030-46147-8_24
State	Published - 2020
Event	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019 - Wurzburg, Germany Duration: 16 Sep 2019 → 20 Sep 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11907 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019
Country/Territory	Germany
City	Wurzburg
Period	16/09/19 → 20/09/19

Keywords

Machine Learning
Neural networks
Sobolev Training

Access to Document

10.1007/978-3-030-46147-8_24

Cite this

Kissel, M., & Diepold, K. (2020). Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks. In U. Brefeld, E. Fromont, A. Hotho, A. Knobbe, M. Maathuis, & C. Robardet (Eds.), Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Proceedings (pp. 399-414). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11907 LNAI). Springer. https://doi.org/10.1007/978-3-030-46147-8_24

Kissel, Matthias ; Diepold, Klaus. / Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Proceedings. editor / Ulf Brefeld ; Elisa Fromont ; Andreas Hotho ; Arno Knobbe ; Marloes Maathuis ; Céline Robardet. Springer, 2020. pp. 399-414 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{a54992173bad4dd8bd4c198e91b0a382,

title = "Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks",

abstract = "With Sobolev Training, neural networks are trained to fit target output values as well as target derivatives with respect to the inputs. This leads to better generalization and fewer required training examples for certain problems. In this paper, we present a training pipeline that enables Sobolev Training for regression problems where target derivatives are not directly available. Thus, we propose to use a least-squares estimate of the target derivatives based on function values of neighboring training samples. We show for a variety of black-box function regression tasks that our training pipeline achieves smaller test errors compared to the traditional training method. Since our method has no additional requirements on the data collection process, it has great potential to improve the results for various regression tasks.",

keywords = "Machine Learning, Neural networks, Sobolev Training",

author = "Matthias Kissel and Klaus Diepold",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2020.; European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019 ; Conference date: 16-09-2019 Through 20-09-2019",

year = "2020",

doi = "10.1007/978-3-030-46147-8_24",

language = "English",

isbn = "9783030461461",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "399--414",

editor = "Ulf Brefeld and Elisa Fromont and Andreas Hotho and Arno Knobbe and Marloes Maathuis and C{\'e}line Robardet",

booktitle = "Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Proceedings",

}

Kissel, M & Diepold, K 2020, Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks. in U Brefeld, E Fromont, A Hotho, A Knobbe, M Maathuis & C Robardet (eds), Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11907 LNAI, Springer, pp. 399-414, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019, Wurzburg, Germany, 16/09/19. https://doi.org/10.1007/978-3-030-46147-8_24

Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks. / Kissel, Matthias; Diepold, Klaus.
Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Proceedings. ed. / Ulf Brefeld; Elisa Fromont; Andreas Hotho; Arno Knobbe; Marloes Maathuis; Céline Robardet. Springer, 2020. p. 399-414 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11907 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks

AU - Kissel, Matthias

AU - Diepold, Klaus

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2020.

PY - 2020

Y1 - 2020

N2 - With Sobolev Training, neural networks are trained to fit target output values as well as target derivatives with respect to the inputs. This leads to better generalization and fewer required training examples for certain problems. In this paper, we present a training pipeline that enables Sobolev Training for regression problems where target derivatives are not directly available. Thus, we propose to use a least-squares estimate of the target derivatives based on function values of neighboring training samples. We show for a variety of black-box function regression tasks that our training pipeline achieves smaller test errors compared to the traditional training method. Since our method has no additional requirements on the data collection process, it has great potential to improve the results for various regression tasks.

AB - With Sobolev Training, neural networks are trained to fit target output values as well as target derivatives with respect to the inputs. This leads to better generalization and fewer required training examples for certain problems. In this paper, we present a training pipeline that enables Sobolev Training for regression problems where target derivatives are not directly available. Thus, we propose to use a least-squares estimate of the target derivatives based on function values of neighboring training samples. We show for a variety of black-box function regression tasks that our training pipeline achieves smaller test errors compared to the traditional training method. Since our method has no additional requirements on the data collection process, it has great potential to improve the results for various regression tasks.

KW - Machine Learning

KW - Neural networks

KW - Sobolev Training

UR - http://www.scopus.com/inward/record.url?scp=85084807612&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-46147-8_24

DO - 10.1007/978-3-030-46147-8_24

M3 - Conference contribution

AN - SCOPUS:85084807612

SN - 9783030461461

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 399

EP - 414

BT - Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Proceedings

A2 - Brefeld, Ulf

A2 - Fromont, Elisa

A2 - Hotho, Andreas

A2 - Knobbe, Arno

A2 - Maathuis, Marloes

A2 - Robardet, Céline

PB - Springer

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019

Y2 - 16 September 2019 through 20 September 2019

ER -

Kissel M, Diepold K. Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks. In Brefeld U, Fromont E, Hotho A, Knobbe A, Maathuis M, Robardet C, editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Proceedings. Springer. 2020. p. 399-414. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-46147-8_24

Sobolev Training with Approximated Derivatives for Black-Box Function Regression with Neural Networks

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this