Effective Version Space Reduction for Convolutional Neural Networks

Jiayu Liu; Ioannis Chiotellis; Rudolph Triebel; Daniel Cremers

doi:10.1007/978-3-030-67661-2_6

Effective Version Space Reduction for Convolutional Neural Networks

Jiayu Liu, Ioannis Chiotellis, Rudolph Triebel, Daniel Cremers

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In active learning, sampling bias could pose a serious inconsistency problem and hinder the algorithm from finding the optimal hypothesis. However, many methods for neural networks are hypothesis space agnostic and do not address this problem. We examine active learning with convolutional neural networks through the principled lens of version space reduction. We identify the connection between two approaches – prior mass reduction and diameter reduction – and propose a new diameter-based querying method – the minimum Gibbs-vote disagreement. By estimating version space diameter and bias, we illustrate how version space of neural networks evolves and examine the realizability assumption. With experiments on MNIST, Fashion-MNIST, SVHN and STL-10 datasets, we demonstrate that diameter reduction methods reduce the version space more effectively and perform better than prior mass reduction and other baselines, and that the Gibbs vote disagreement is on par with the best query method.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2020, Proceedings
Editors	Frank Hutter, Kristian Kersting, Jefrey Lijffijt, Isabel Valera
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	85-100
Number of pages	16
ISBN (Print)	9783030676605
DOIs	https://doi.org/10.1007/978-3-030-67661-2_6
State	Published - 2021
Event	European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2020 - Virtual, Online Duration: 14 Sep 2020 → 18 Sep 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12458 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2020
City	Virtual, Online
Period	14/09/20 → 18/09/20

Keywords

Active learning
Deep learning
Diameter reduction
Version space

Access to Document

10.1007/978-3-030-67661-2_6

Cite this

Liu, J., Chiotellis, I., Triebel, R., & Cremers, D. (2021). Effective Version Space Reduction for Convolutional Neural Networks. In F. Hutter, K. Kersting, J. Lijffijt, & I. Valera (Eds.), Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2020, Proceedings (pp. 85-100). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12458 LNAI). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-67661-2_6

Liu, Jiayu ; Chiotellis, Ioannis ; Triebel, Rudolph et al. / Effective Version Space Reduction for Convolutional Neural Networks. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2020, Proceedings. editor / Frank Hutter ; Kristian Kersting ; Jefrey Lijffijt ; Isabel Valera. Springer Science and Business Media Deutschland GmbH, 2021. pp. 85-100 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{b56dc68792f443efa4ac69f92b5ee655,

title = "Effective Version Space Reduction for Convolutional Neural Networks",

abstract = "In active learning, sampling bias could pose a serious inconsistency problem and hinder the algorithm from finding the optimal hypothesis. However, many methods for neural networks are hypothesis space agnostic and do not address this problem. We examine active learning with convolutional neural networks through the principled lens of version space reduction. We identify the connection between two approaches – prior mass reduction and diameter reduction – and propose a new diameter-based querying method – the minimum Gibbs-vote disagreement. By estimating version space diameter and bias, we illustrate how version space of neural networks evolves and examine the realizability assumption. With experiments on MNIST, Fashion-MNIST, SVHN and STL-10 datasets, we demonstrate that diameter reduction methods reduce the version space more effectively and perform better than prior mass reduction and other baselines, and that the Gibbs vote disagreement is on par with the best query method.",

keywords = "Active learning, Deep learning, Diameter reduction, Version space",

author = "Jiayu Liu and Ioannis Chiotellis and Rudolph Triebel and Daniel Cremers",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2020 ; Conference date: 14-09-2020 Through 18-09-2020",

year = "2021",

doi = "10.1007/978-3-030-67661-2_6",

language = "English",

isbn = "9783030676605",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "85--100",

editor = "Frank Hutter and Kristian Kersting and Jefrey Lijffijt and Isabel Valera",

booktitle = "Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2020, Proceedings",

}

Liu, J, Chiotellis, I, Triebel, R & Cremers, D 2021, Effective Version Space Reduction for Convolutional Neural Networks. in F Hutter, K Kersting, J Lijffijt & I Valera (eds), Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12458 LNAI, Springer Science and Business Media Deutschland GmbH, pp. 85-100, European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2020, Virtual, Online, 14/09/20. https://doi.org/10.1007/978-3-030-67661-2_6

Effective Version Space Reduction for Convolutional Neural Networks. / Liu, Jiayu; Chiotellis, Ioannis; Triebel, Rudolph et al.
Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2020, Proceedings. ed. / Frank Hutter; Kristian Kersting; Jefrey Lijffijt; Isabel Valera. Springer Science and Business Media Deutschland GmbH, 2021. p. 85-100 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12458 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Effective Version Space Reduction for Convolutional Neural Networks

AU - Liu, Jiayu

AU - Chiotellis, Ioannis

AU - Triebel, Rudolph

AU - Cremers, Daniel

PY - 2021

Y1 - 2021

N2 - In active learning, sampling bias could pose a serious inconsistency problem and hinder the algorithm from finding the optimal hypothesis. However, many methods for neural networks are hypothesis space agnostic and do not address this problem. We examine active learning with convolutional neural networks through the principled lens of version space reduction. We identify the connection between two approaches – prior mass reduction and diameter reduction – and propose a new diameter-based querying method – the minimum Gibbs-vote disagreement. By estimating version space diameter and bias, we illustrate how version space of neural networks evolves and examine the realizability assumption. With experiments on MNIST, Fashion-MNIST, SVHN and STL-10 datasets, we demonstrate that diameter reduction methods reduce the version space more effectively and perform better than prior mass reduction and other baselines, and that the Gibbs vote disagreement is on par with the best query method.

AB - In active learning, sampling bias could pose a serious inconsistency problem and hinder the algorithm from finding the optimal hypothesis. However, many methods for neural networks are hypothesis space agnostic and do not address this problem. We examine active learning with convolutional neural networks through the principled lens of version space reduction. We identify the connection between two approaches – prior mass reduction and diameter reduction – and propose a new diameter-based querying method – the minimum Gibbs-vote disagreement. By estimating version space diameter and bias, we illustrate how version space of neural networks evolves and examine the realizability assumption. With experiments on MNIST, Fashion-MNIST, SVHN and STL-10 datasets, we demonstrate that diameter reduction methods reduce the version space more effectively and perform better than prior mass reduction and other baselines, and that the Gibbs vote disagreement is on par with the best query method.

KW - Active learning

KW - Deep learning

KW - Diameter reduction

KW - Version space

UR - http://www.scopus.com/inward/record.url?scp=85103236848&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-67661-2_6

DO - 10.1007/978-3-030-67661-2_6

M3 - Conference contribution

AN - SCOPUS:85103236848

SN - 9783030676605

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 85

EP - 100

BT - Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2020, Proceedings

A2 - Hutter, Frank

A2 - Kersting, Kristian

A2 - Lijffijt, Jefrey

A2 - Valera, Isabel

PB - Springer Science and Business Media Deutschland GmbH

T2 - European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2020

Y2 - 14 September 2020 through 18 September 2020

ER -

Liu J, Chiotellis I, Triebel R , Cremers D. Effective Version Space Reduction for Convolutional Neural Networks. In Hutter F, Kersting K, Lijffijt J, Valera I, editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2020, Proceedings. Springer Science and Business Media Deutschland GmbH. 2021. p. 85-100. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-67661-2_6

Effective Version Space Reduction for Convolutional Neural Networks

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this