Three Operator Splitting with Subgradients, Stochastic Gradients, and Adaptive Learning Rates

Alp Yurtsever, Alex Gu, Suvrit Sra

Publikation: Beitrag in Buch/Bericht/KonferenzbandKonferenzbeitragBegutachtung

4 Zitate (Scopus)

Abstract

Three Operator Splitting (TOS) (Davis & Yin, 2017) can minimize the sum of multiple convex functions effectively when an efficient gradient oracle or proximal operator is available for each term. This requirement often fails in machine learning applications: (i) instead of full gradients only stochastic gradients may be available; and (ii) instead of proximal operators, using subgradients to handle complex penalty functions may be more efficient and realistic. Motivated by these concerns, we analyze three potentially valuable extensions of TOS. The first two permit using subgradients and stochastic gradients, and are shown to ensure a O(1/ √ t) convergence rate. The third extension ADAPTOS endows TOS with adaptive stepsizes. For the important setting of optimizing a convex loss over the intersection of convex sets ADAPTOS attains universal convergence rates, i.e., the rate adapts to the unknown smoothness degree of the objective function. We compare our proposed methods with competing methods on various applications.

OriginalspracheEnglisch
TitelAdvances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Redakteure/-innenMarc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Herausgeber (Verlag)Neural information processing systems foundation
Seiten19743-19756
Seitenumfang14
ISBN (elektronisch)9781713845393
PublikationsstatusVeröffentlicht - 2021
Extern publiziertJa
Veranstaltung35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online
Dauer: 6 Dez. 202114 Dez. 2021

Publikationsreihe

NameAdvances in Neural Information Processing Systems
Band24
ISSN (Print)1049-5258

Konferenz

Konferenz35th Conference on Neural Information Processing Systems, NeurIPS 2021
OrtVirtual, Online
Zeitraum6/12/2114/12/21

Fingerprint

Untersuchen Sie die Forschungsthemen von „Three Operator Splitting with Subgradients, Stochastic Gradients, and Adaptive Learning Rates“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren